{"id":20956,"date":"2018-11-06T05:31:55","date_gmt":"2018-11-06T05:31:55","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/vcblog\/?p=20956"},"modified":"2019-02-18T17:47:35","modified_gmt":"2019-02-18T17:47:35","slug":"exploring-clang-tooling-part-3-rewriting-code-with-clang-tidy","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/cppblog\/exploring-clang-tooling-part-3-rewriting-code-with-clang-tidy\/","title":{"rendered":"Exploring Clang Tooling Part 3: Rewriting Code with clang-tidy"},"content":{"rendered":"

In the previous post<\/a> in this series, we used clang-query<\/tt> to examine the Abstract Syntax Tree of a simple source code file. Using clang-query<\/tt>, we can prototype an AST Matcher which we can use in a clang-tidy<\/tt> check to refactor code in bulk.<\/p>\n

This time, we will complete the rewriting of the source code.<\/p>\n

\"\"<\/a><\/p>\n

Let’s return to MyFirstCheck.cpp we generated earlier<\/a> and update the registerMatchers<\/tt> method. First we can refactor it to port both function declarations and function calls, using the callExpr()<\/tt> and callee()<\/tt> matchers we used in the previous post:<\/p>\n

void MyFirstCheckCheck::registerMatchers(MatchFinder *Finder) {\r\n    \r\n  auto nonAwesomeFunction = functionDecl(\r\n    unless(matchesName(\"^::awesome_\"))\r\n    );\r\n\r\n  Finder->addMatcher(\r\n    nonAwesomeFunction.bind(\"addAwesomePrefix\")\r\n    , this);\r\n\r\n  Finder->addMatcher(\r\n    callExpr(callee(nonAwesomeFunction)).bind(\"addAwesomePrefix\")\r\n    , this);\r\n}<\/pre>\n
Because Matchers are really C++ code, we can extract them into variables and compose them into multiple other Matchers, as done here with nonAwesomeFunction<\/tt>.<\/p>\n
In this case, I have narrowed the declaration matcher to match only on function declarations which do not start with awesome_<\/tt>. That matcher is then used once with a binder addAwesomePrefix<\/tt>, then again to specify the callee()<\/tt> of a callExpr()<\/tt>, again binding the relevant expression to the name addAwesomePrefix<\/tt>.<\/p>\n
Because large scale refactoring often involves primarily changing particular expressions, it generally makes sense to separately define the matchers for the declaration to match and the expressions referencing those declarations. In my experience, the matchers for declarations can get complicated for example with exclusions due to limitations of a reflection system, or with more specifics about functions with particular return types or argument types. Centralizing those cases helps keep your refactoring code maintainable.<\/p>\n
Another change I have made is that I renamed the binding from x<\/tt> to addAwesomePrefix<\/tt>. This is notable because it uses verbs to describe what should be done with the matches. It should be clear from reading matcher bindings what the result of invoking the fix is to be. Binding names can then be seen as a weakly-typed string-based language interface between the matcher and the replacement code.\nWe can then implement MyFirstCheckCheck::check<\/tt> to consume the bindings. A first approximation might look like:<\/p>\n
void MyFirstCheckCheck::check(const MatchFinder::MatchResult &Result) {\r\n  if (const auto MatchedDecl = Result.Nodes.getNodeAs<FunctionDecl>(\"addAwesomePrefix\"))\r\n  {\r\n    diag(MatchedDecl->getLocation(), \"function is insufficiently awesome\")\r\n      << FixItHint::CreateInsertion(MatchedDecl->getLocation(), \"awesome_\");\r\n  }\r\n\r\n  if (const auto MatchedExpr = Result.Nodes.getNodeAs<CallExpr>(\"addAwesomePrefix\"))\r\n  {\r\n    diag(MatchedExpr->getExprLoc(), \"code is insufficiently awesome\")\r\n      << FixItHint::CreateInsertion(MatchedExpr->getExprLoc(), \"awesome_\");\r\n  }\r\n}<\/pre>\n
Perhaps a better implementation would reduce the duplication of the diagnostic code:<\/p>\n
void MyFirstCheckCheck::check(const MatchFinder::MatchResult &Result) {\r\n  SourceLocation insertionLocation;\r\n  if (const auto MatchedDecl = Result.Nodes.getNodeAs<FunctionDecl>(\"addAwesomePrefix\"))\r\n  {\r\n    insertionLocation = MatchedDecl->getLocation();\r\n  } else if (const auto MatchedExpr = Result.Nodes.getNodeAs<CallExpr>(\"addAwesomePrefix\"))\r\n  {\r\n    insertionLocation = MatchedExpr->getExprLoc();\r\n  }\r\n  diag(insertionLocation, \"code is insufficiently awesome\")\r\n      << FixItHint::CreateInsertion(insertionLocation, \"awesome_\");\r\n}<\/pre>\n
Because the FunctionDecl<\/tt> and the CallExpr<\/tt> do not share an inheritance hierarchy, we need separate casting conditions for each. Even if they did share an inheritance hierarchy, we need to call getLocation<\/tt> in one case, and getExprLoc<\/tt> in another. The reason for that is that Clang records many relevant locations for each AST node. The developer of the clang-tidy check needs to know which location accessor method is appropriate or required for each situation.\nA further improvement is to change the casts to accept the relevant types of FunctionDecl<\/tt> and CallExpr<\/tt> – NamedDecl<\/tt> and Expr<\/tt> respectively.<\/p>\n
if (const auto MatchedDecl = Result.Nodes.getNodeAs<NamedDecl>(\"addAwesomePrefix\"))\r\n{\r\n  insertionLocation = MatchedDecl->getLocation();\r\n} else if (const auto MatchedExpr = Result.Nodes.getNodeAs<Expr>(\"addAwesomePrefix\"))\r\n{\r\n  insertionLocation = MatchedExpr->getExprLoc();\r\n}<\/pre>\n
This change enforces the idea that the names of bound nodes form a weakly-typed interface between the Matcher code and the Rewriter code. Because the Rewriter code now expects the addAwesomePrefix<\/tt> to be used with the base types NamedDecl<\/tt> and Expr<\/tt>, other Matcher code can take advantage of that. We can now re-use the addAwesomePrefix<\/tt> binding name to add a prefix to field declarations or member expressions for example because their corresponding Clang AST classes also inherit NamedDecl<\/tt>:<\/p>\n
auto nonAwesomeField = fieldDecl(unless(hasName(\"::awesome_\")));\r\nFinder->addMatcher(\r\n  nonAwesomeField.bind(\"addAwesomePrefix\")\r\n  , this);\r\n\r\nFinder->addMatcher(\r\n  memberExpr(member(nonAwesomeField)).bind(\"addAwesomePrefix\")\r\n  , this);<\/pre>\n
Notice that this code is comparable to the matchers we wrote for the functionDecl<\/tt>\/callExpr<\/tt> pairing. Taking advantage of the binding name interface, we can continue extending our matcher code to port variable declarations without changing the rewriter side of that interface:<\/p>\n
void MyFirstCheckCheck::registerMatchers(MatchFinder *Finder) {\r\n  \r\n  auto nonAwesome = namedDecl(\r\n    unless(matchesName(\"::awesome_.*\"))\r\n    );\r\n\r\n  auto nonAwesomeFunction = functionDecl(nonAwesome);\r\n  \/\/ void foo(); \r\n  Finder->addMatcher(\r\n    nonAwesomeFunction.bind(\"addAwesomePrefix\")\r\n    , this);\r\n\r\n  \/\/ foo();\r\n  Finder->addMatcher(\r\n    callExpr(callee(nonAwesomeFunction)).bind(\"addAwesomePrefix\")\r\n    , this);\r\n\r\n  auto nonAwesomeVar = varDecl(nonAwesome);\r\n  \/\/ int foo;\r\n  Finder->addMatcher(\r\n    nonAwesomeVar.bind(\"addAwesomePrefix\")\r\n    , this);\r\n\r\n  \/\/ foo = 7;\r\n  Finder->addMatcher(\r\n    declRefExpr(to(nonAwesomeVar)).bind(\"addAwesomePrefix\")\r\n    , this);\r\n\r\n  auto nonAwesomeField = fieldDecl(nonAwesome);\r\n  \/\/ int m_foo;\r\n  Finder->addMatcher(\r\n    nonAwesomeField.bind(\"addAwesomePrefix\")\r\n    , this);\r\n\r\n  \/\/ m_foo = 42;\r\n  Finder->addMatcher(\r\n    memberExpr(member(nonAwesomeField)).bind(\"addAwesomePrefix\")\r\n    , this);\r\n}<\/pre>\n
Location Location Location<\/h2>\n
Let’s return to the check<\/tt> implementation and examine it. This method is responsible for implementing the rewriting of the source code as described by the matchers and their bound nodes.\nIn this case, we have inserted code at the SourceLocation<\/tt> returned by either getLocation()<\/tt> or getExprLoc()<\/tt> of NamedDecl<\/tt> or Expr<\/tt> respectively. Clang AST classes have many methods returning SourceLocation<\/tt> which refer to various places in the source code related to particular AST nodes.\nFor example, the CallExpr<\/tt><\/a> has SourceLocation<\/tt> accessors getBeginLoc<\/tt>, getEndLoc<\/tt> and getExprLoc<\/tt>. It is currently difficult to discover how a particular position in the source code relates to a particular SourceLocation<\/tt> accessor.<\/p>\n
clang::VarDecl<\/tt> represents variable declarations in the Clang AST. clang::ParmVarDecl<\/tt> inherits clang::VarDecl<\/tt> and represents parameter declarations. Notice that in all cases, end<\/tt> locations indicate the beginning of the last token, not the end of it. Note also that in the second example below, the source locations of the call used to initialize the variable are not part of the variable. It is necessary to traverse to the initialization expression to access those.<\/p>\n
\"\"<\/a><\/p>\n
clang::FunctionDecl<\/tt> represents function declarations in the Clang AST. clang::CXXMethodDel<\/tt> inherits clang::FunctionDecl<\/tt> and represents method declarations. Note that the location of the return type is not always given by getBeginLoc<\/tt> in C++.<\/p>\n
\"\"<\/a><\/p>\n
clang::CallExpr<\/tt> represents function calls in the Clang AST. clang::CXXMemberCallExpr<\/tt> inherits clang::CallExpr<\/tt> and represents method calls. Note that when calling free functions (represented by a clang::CallExpr<\/tt>), the getExprLoc<\/tt> and the getBeginLoc<\/tt> will be the same. Always chose the semantically correct location accessor, rather than a location which appears to indicate the correct position.<\/p>\n
\"\"<\/a><\/p>\n
It is important to know that locations on AST classes point to the start of tokens in all cases. This can be initially confusing when examining end locations. Sometimes to get to a desired location, it is necessary to use getLocWithOffset()<\/tt> to advance or retreat a SourceLocation<\/tt>. Advancing to the end of a token can be achieved with Lexer::getLocForEndOfToken<\/tt>.<\/p>\n
The source code locations of arguments to the function call are not accessible from the CallExpr<\/tt>, but must be accessed via AST nodes for the arguments themselves.<\/p>\n
\/\/ Get the zeroth argument:\r\nExpr* arg0 = someCallExpr->getArg(0);\r\nSourceLocation arg0Loc = arg0->getExprLoc();<\/pre>\n
Every AST node has accessors getBeginLoc<\/tt> and getEndLoc<\/tt>. Expression nodes additionally have a getExprLoc<\/tt>, and declaration nodes have an additional getLocation<\/tt> accessor. More-specific subclasses have more-specific accessors for locations relevant to the C++ construct they represent. Source code locations in Clang are comprehensive, but accessing them can get complex as requirements become more advanced. A future blog post may explore this topic in more detail if there is interest among the readership.<\/p>\n
Once we have acquired the locations we are interested in, we need to insert, remove or replace source code fragments at those locations.<\/p>\n
Let’s return to MyFirstCheck.cpp:<\/p>\n
diag(insertionLocation, \"code is insufficiently awesome\")\r\n    << FixItHint::CreateInsertion(insertionLocation, \"awesome_\");<\/pre>\n
diag<\/tt> is a method on the ClangTidyCheck<\/tt> base class<\/a>. The purpose of it is to issue diagnostics and messages to the user. It can be called with just a source location and a message, causing a diagnostic to be emitted at the specified location:<\/p>\n
diag(insertionLocation, \"code is insufficiently awesome\");<\/pre>\n
Resulting in:<\/p>\n
    testfile.cpp:19:5: warning: code is insufficiently awesome [misc-my-first-check]\r\n    int addTwo(int num)\r\n        ^\r\n<\/pre>\n
The diag<\/tt> method returns a DiagnosticsBuilder<\/tt> to which we can stream fix suggestions using FixItHint<\/tt><\/a>.<\/p>\n
The CreateRemoval<\/tt> method creates a FixIt<\/tt> for removal of a range of source code. At its heart, a SourceRange<\/tt> is just a pair of SourceLocation<\/tt>s. If we wanted to remove the awesome_<\/tt> prefix from functions which have it, we might expect to write something like this:<\/p>\n
void MyFirstCheckCheck::registerMatchers(MatchFinder *Finder) {\r\n  \r\n  Finder->addMatcher(\r\n    functionDecl(\r\n      matchesName(\"::awesome_.*\")\r\n      ).bind(\"removeAwesomePrefix\")\r\n    , this);\r\n}\r\n\r\nvoid MyFirstCheckCheck::check(const MatchFinder::MatchResult &Result) {\r\n\r\n  if (const auto MatchedDecl = Result.Nodes.getNodeAs<NamedDecl>(\"removeAwesomePrefix\"))\r\n  {\r\n      auto removalStartLocation = MatchedDecl->getLocation();\r\n      auto removalEndLocation = removalStartLocation.getLocWithOffset(sizeof(\"awesome_\") - 1);\r\n      auto removalRange = SourceRange(removalStartLocation, removalEndLocation);\r\n\r\n      diag(removalStartLocation, \"code is too awesome\")\r\n          << FixItHint::CreateRemoval(removalRange);\r\n  }\r\n}<\/pre>\n
The matcher part of this code is fine, but when we run clang-tidy, we find that the removal is applied to the entire function name, not only the awesome_<\/tt> prefix. The problem is that Clang extends the end of the removal range to the end of the token pointed to by the end. This is symmetric with the fact that AST nodes have getEndLoc()<\/tt> methods which point to the start of the last token. Usually, the intent is to remove or replace entire tokens.<\/p>\n
To make a replacement or removal in source code which extends into the middle of a token, we need to indicate that we are replacing a range of characters instead of a range of tokens, using CharSourceRange::getCharRange<\/tt>:<\/p>\n
auto removalRange = CharSourceRange::getCharRange(removalStartLocation, removalEndLocation);\r\n<\/pre>\n
Conclusion<\/h2>\n
This concludes the mini-series about writing clang-tidy<\/tt> checks. This series has been an experiment to gauge interest, and there is a lot more content to cover in further posts if there is interest among the readership.<\/p>\n
Further topics can cover topics that occur in the real world such as<\/p>\n