{"id":23891,"date":"2018-11-27T12:12:01","date_gmt":"2018-11-27T12:12:01","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/cppblog\/?p=23891"},"modified":"2019-02-22T12:18:29","modified_gmt":"2019-02-22T12:18:29","slug":"exploring-clang-tooling-using-build-tools-with-clang-tidy","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/cppblog\/exploring-clang-tooling-using-build-tools-with-clang-tidy\/","title":{"rendered":"Exploring Clang Tooling – Using Build Tools with clang-tidy"},"content":{"rendered":"

This post is part of a regular series of posts where the C++ product team and other guests answer questions we have received from customers. The questions can be about anything C++ related: MSVC toolset, the standard language and library, the C++ standards committee, isocpp.org, CppCon, etc.<\/em><\/p>\n

Today\u2019s post is by guest author Stephen Kelly, who is a developer at Havok, a contributor to Qt and CMake and a blogger<\/a>. This post is part of a series where he is sharing his experience using Clang tooling in his current team.<\/em><\/p>\n

The previous series about clang-tidy<\/tt><\/a> on this blog covered the basics of creating a clang-tidy<\/tt> extension and tooling to support that in the form of clang-query<\/tt>.<\/p>\n

While the series focused on single-file examples for simplicity, developers progressing in this direction will need to run the tooling on all of the files in their project at once, or on all files which match a specific pattern.<\/p>\n

Delayed refactoring<\/h3>\n

The first problem with processing multiple files is that we can no longer change files as we process them and discover locations to refactor. Tools like clang-tidy<\/tt> only work if the source code compiles, so a process which changed a header file while processing the first source file would cause the next source file to not be compilable.<\/p>\n

\"\"<\/p>\n

To resolve this problem, clang-tidy<\/tt> has the ability to export refactoring changes to a .yaml<\/tt> file, instead of changing the files directly.<\/p>\n

\"\"<\/p>\n

The clang-apply-replacements<\/tt> tool can then be run on a directory of .yaml<\/tt> files in order to apply the changes to all of the files at once.<\/p>\n

\"\"<\/a><\/p>\n

The run-clang-tidy<\/tt><\/a> script in the clang repository helps with these tasks. It accepts a pattern of files and processes all matching files in parallel, making use of all available cores.<\/p>\n

Build tools<\/h3>\n

Consider the similarity between using a compiler with a .cpp<\/tt> file to produce an object file and using clang-tidy<\/tt> to produce a .yaml<\/tt> file.<\/p>\n

\"\"<\/p>\n

This similarity implies that we can use build tools with clang-tidy<\/tt>.<\/p>\n

We can use any tool to generate a Ninja buildsystem, but generally they are not currently optimized for generating commands which invoke clang-tidy instead of a compiler. Although CMake has clang-tidy<\/tt> support<\/a>, it doesn’t have direct support for delayed refactoring, so the CMake integration is currently more suitable to linting instead of refactoring tasks.<\/p>\n

For now, using some tricks, we can use CMake to generate a buildsystem<\/a> from a compile_commands.json<\/tt> file. The generated ‘buildsystem’ simply uses clang-tidy<\/tt> in place of the compiler, so that it outputs .yaml<\/tt> files instead of object files. The CMake script produces a ‘buildsystem’ based on the content of a compile_commands.json<\/tt> file which you have already generated.<\/p>\n

We can instruct CMake to generate a Ninja ‘buildsystem’ and run a ‘build’ in the normal way to invoke the refactor:<\/p>\n

cmake .. -G Ninja -DCMAKE_CXX_COMPILER=<path_to_clang_tidy>\r\ncmake --build .\r\n<\/pre>\n
Ninja processes the inputs in parallel, so this results in a collection of .yaml<\/tt> files in the fixes<\/tt> directory. We can use clang-apply-replacements<\/tt> to apply those fixes to the source code.<\/p>\n
Using CMake and Ninja brings advantages that the run-clang-tidy<\/tt> script doesn’t provide. Because we are modelling mechanical refactoring as a build task, we can use other build tools which work with Ninja and CMake. To start, we can convert the log of Ninja<\/a> performing the refactor to a trace which is compatible with the Chrome about:tracing<\/tt> tool. This gives output showing the length of time taken for each translation unit:<\/p>\n
\"\"<\/a><\/p>\n
We can also take advantage of the fact that we are now using CMake to handle the refactoring. Using Visual Studio Code and the CMake Tools<\/a> plugin, we can simply open the folder containing the CMakeLists.txt and trigger the refactoring task from there.<\/p>\n
Add a custom kit<\/a> to the CMake Tools<\/tt> for running clang-tidy<\/tt>:<\/p>\n
{\r\n  \"name\": \"Clang tidy\",\r\n  \"compilers\": {\r\n    \"CXX\": \"C:\/dev\/prefix\/bin\/clang-tidy.exe\"\r\n  }\r\n}\r\n<\/pre>\n
Now, when we invoke Build<\/tt> in Visual Studio Code, the refactoring is started. Diagnostics are also collected with easy navigation to the source code.<\/p>\n
\"\"<\/p>\n
Because CMake can generate Visual Studio solutions, it is also possible to control the refactoring from within Visual Studio. As this requires creating a Toolset file<\/a> to replace the compiler with clang-tidy<\/tt>, it is slightly out of scope of this post but it follows the same pattern to achieve the result.<\/p>\n
Distributing the refactor<\/h3>\n
Consider how we distribute our build tasks on the network.<\/p>\n
\"\"<\/p>\n
If we treat clang-tidy<\/tt> as a compiler, then we should be able to use a build-distribution tool to distribute our refactoring task on the network.<\/p>\n
\"\"<\/p>\n
One such build distribution tool is Icecream<\/a>, which is popular on Linux systems and available under the GPL. Icecream works by sending an archive of the build tools to client machines so that the actual compilation is run on the remote machine and the resulting object file is sent back to the client.<\/p>\n
By packaging the clang-tidy executable<\/a>, renamed to clang<\/tt> so that Icecream accepts it, we can refactor on remote machines and send resulting .obj files (named so that Icecream accepts them, but containing yaml content) to clients. The Icecream Monitor tool then shows a progress of the distributed task among the build nodes.<\/p>\n
\"\"<\/p>\n
This work distribution brings a significant increase in speed to the refactoring task. Using this technique I have been able to make mechanical changes to the LLVM\/Clang source (millions of lines of code) in minutes which would otherwise take hours if run only locally. Because there is no need to link libraries while refactoring, each refactor does not conflict with any other and the process can be embarrassingly parallel.<\/p>\n
Conclusion<\/h3>\n
Mechanical refactoring with clang-tidy<\/tt> requires distribution over a network in order to complete in reasonable time on large codebases. What other build tools do you think would be adaptable for refactoring tasks? Let us know in the comments below or contact the author directly via e-mail at stkelly@microsoft.com<\/a>, or on Twitter @steveire<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"
This post is part of a regular series of posts where the C++ product team and other guests answer questions we have received from customers. The questions can be about anything C++ related: MSVC toolset, the standard language and library, the C++ standards committee, isocpp.org, CppCon, etc. Today\u2019s post is by guest author Stephen Kelly, […]<\/p>\n","protected":false},"author":890,"featured_media":23892,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-23891","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cplusplus"],"acf":[],"blog_post_summary":"
This post is part of a regular series of posts where the C++ product team and other guests answer questions we have received from customers. The questions can be about anything C++ related: MSVC toolset, the standard language and library, the C++ standards committee, isocpp.org, CppCon, etc. Today\u2019s post is by guest author Stephen Kelly, […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/posts\/23891","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/users\/890"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/comments?post=23891"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/posts\/23891\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/media\/23892"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/media?parent=23891"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/categories?post=23891"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/tags?post=23891"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}