{"id":2243,"date":"2010-03-30T03:11:00","date_gmt":"2010-03-30T03:11:00","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/visualstudio\/2010\/03\/30\/wpf-in-visual-studio-2010-part-6-automated-ui-testing\/"},"modified":"2022-10-14T12:56:00","modified_gmt":"2022-10-14T19:56:00","slug":"wpf-in-visual-studio-2010-part-6-automated-ui-testing","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/visualstudio\/wpf-in-visual-studio-2010-part-6-automated-ui-testing\/","title":{"rendered":"WPF in Visual Studio 2010 \u2013 Part 6 : Automated UI Testing"},"content":{"rendered":"<p>This is the sixth article in the <a href=\"https:\/\/devblogs.microsoft.com\/visualstudio\/wpf-in-visual-studio-2010-part-1-introduction\/\">WPF in Visual Studio 2010 series<\/a>. This week, guest author Phil Price gives us his view of what it took to test the new WPF UI. <em>&#8211; Paul Harrington<\/em><\/p>\n<h3>Background<\/h3>\n<p>This post covers an overview of techniques that we used to create and maintain automated user interface regression tests for Visual Studio. Regression tests are a type of software test that, collectively, aim to be an oracle of expected functionality for the target application, run often against new builds of product \u2013 they aim to uncover regressions in behavior introduced in a new build.<\/p>\n<h3>Pre-Visual Studio 2010 UI Automation<\/h3>\n<p>Throughout Visual Studio 2005 and Visual Studio 2008 we built a sizable UI automation framework that abstracted the different facets of the user interface into atomic classes. A concrete example of this would be a class named \u201cToolWindow\u201d which would have methods such as \u201cClickClose\u201d which would click the \u201cX\u201d button on the tool window to close it, and \u201cDragTo(x, y)\u201d which would drag a tool window to the specified screen coordinate. Using this framework a vast suite of regression tests was built to exercise the functionality within Visual Studio. Prior to Visual Studio 2010, the UI automation framework would largely use <a href=\"http:\/\/msdn.microsoft.com\/en-us\/library\/aa383751(VS.85).aspx\">HWND<\/a> search techniques to find a particular HWND of interest. A trivial example would be to find the status bar and get its current caption*:<\/p>\n<blockquote><p>\/\/ Find visual studio main window\nIntPtr visualStudioHwnd = NativeMethods.FindWindowEx(\nIntPtr.Zero, IntPtr.Zero, &#8220;wndclass_desked_gsk&#8221;, null);<\/p>\n<p>\/\/ Find the Status Bar\nIntPtr statusBar = NativeMethods.FindWindowEx(\nvisualStudioHwnd, IntPtr.Zero, &#8220;VsStatusBar&#8221;, null);<\/p>\n<p>\/\/ Wraps SendMessage of WM_GETTEXTLENGTH and WM_GETTEXT\nstring statusBarText = NativeMethods.GetWindowText(statusBar);<\/p><\/blockquote>\n<p>&nbsp;<\/p>\n<p>* Please note that these examples are designed to <i>illustrate<\/i> the APIs we use to achieve UI automation. A test case would not do this directly, but go through a framework.<\/p>\n<p>For some controls HWNDs wouldn\u2019t expose enough information. In this case, one could retrieve the <a href=\"http:\/\/msdn.microsoft.com\/en-us\/library\/ms971310.aspx\">MSAA<\/a> <a href=\"http:\/\/msdn.microsoft.com\/en-us\/library\/dd318466(VS.85).aspx\">IAccessible<\/a> from the HWND and party on that. IAccessible is the same interface that screen readers and other assistive technology may use to gather information about an application\u2019s user interface and capabilities. It just so happens that it provides great information for consumption from UI tests too. An example of this would be to click each of the visible top level menu items.<\/p>\n<blockquote>\n<pre class=\"code\">IntPtr dock = NativeMethods.FindWindowEx(\r\n    visualStudioHwnd, IntPtr.Zero, null, \"MsoDockTop\");\r\nIntPtr mainMenu = NativeMethods.FindWindowEx(\r\n    dock, IntPtr.Zero, null, \"Menu Bar\");\r\n\/\/ Get the IAccessible.\r\n\/\/ Wraps ole32!AccessibleFromWindow\r\nIAccessible menuBarAccessible = NativeMethods.GetAccessibleFromWindow(\r\n    mainMenu, (uint)NativeMethods.OBJID.CLIENT);\r\n\/\/ Wraps ole32!AccessibleChildren\r\nobject[] children = NativeMethods.GetAccessibleChildren(menuBarAccessible);\r\nforeach (IAccessible child in children\r\n                .Where(child =&gt; child is IAccessible).Cast&lt;IAccessible&gt;())\r\n{\r\n    \/\/ Skip invisible children (hidden)\r\n    if ((uint)child.accState == ((uint)child.accState | NativeMethods.STATE_SYSTEM_INVISIBLE))\r\n    {\r\n        continue;\r\n    }\r\n    \/\/\/\/ Click each item\r\n    int top, left, width, height = 0;\r\n    child.accLocation(out left, out top, out width, out height, 0);\r\n    NativeMethods.SetCursorPos(left + width \/ 2, top + height \/ 2);\r\n    NativeMethods.SendInput(\r\n        NativeMethods.MOUSEINPUT.FromFlags(NativeMethods.MOUSEEVENTF_LEFTDOWN));\r\n    NativeMethods.SendInput(\r\n        NativeMethods.MOUSEINPUT.FromFlags(NativeMethods.MOUSEEVENTF_LEFTUP));\r\n    Thread.Sleep(100);\r\n}<\/pre>\n<\/blockquote>\n<p>&nbsp;<\/p>\n<p>By the end of the Visual Studio 2008 product cycle, HWND searching, various User32 calls and IAccessible were the techniques used to execute our UI automation. It had been successful and we\u2019d shipped two products using it. But, as you\u2019ve probably already guessed, that was about to change.<\/p>\n<h3>Intermezzo: Numbers<\/h3>\n<p>Before we move on to UI testing for Visual Studio 2010, I\u2019d like to share a few numbers that represent the scale of automated UI regression testing for Visual Studio 2008.<\/p>\n<ul>\n<li>Visual Studio 2008 had <b>~65,000<\/b> automated UI regression tests at the time of shipping.<\/li>\n<li>The core UI automation framework (non-Visual Studio specifics) was around <b>~41,000 <\/b>lines of code (excluding comments, <b>175 <\/b>files)<\/li>\n<li>The core Visual Studio parts (think tool windows, editors, projects etc.) of the UI automation framework was around ~<b>147,000 <\/b>lines of code (excluding comments, <b>763<\/b> files)<\/li>\n<li>On top of the core Visual Studio parts was built an additional layer of product features (Data tools, Debugger, Deployment, DLinq, Test Tools, Office Tools, VSSDK, Smart Devices, Team Foundation, VC, Web Forms, Win Forms and Class Designer) totaling an additional <b>~400,000<\/b> lines of code (excluding comments, <b>1840<\/b> files)<\/li>\n<\/ul>\n<h3>Visual Studio 2010 UI Automation<\/h3>\n<p>With the introduction of the WPF shell the previously used technique of zipping around the HWND tree became more difficult, along with calls to APIs like <a href=\"http:\/\/msdn.microsoft.com\/en-us\/library\/ms644950(VS.85).aspx\">SendMessage<\/a> and <a href=\"http:\/\/msdn.microsoft.com\/en-us\/library\/ms633584(VS.85).aspx\">GetWindowLong<\/a> which were used quite extensively. There are still HWNDs within the WPF shell for some content \u2013 but many HWNDs were removed in place of WPF UI elements, and the structure up to the HWND you may care about (for example Solution Explorer) changed. We had built up assumptions about the structure of the HWND tree within the UI automation framework for Visual Studio. These assumptions no longer held true.<\/p>\n<p>So what to do? It\u2019s clear that relying on a HWND structure isn\u2019t going to cut it, but we\u2019d like to continue to use the majority of our old regression tests from the previous release of Visual Studio because it would be very expensive to re-write them all.<\/p>\n<p>Enter <a href=\"http:\/\/msdn.microsoft.com\/en-us\/library\/ms747327.aspx\">Microsoft UI Automation<\/a>, aka UIA. UIA is, to quote MSDN, <i>\u201cthe new accessibility framework for Microsoft Windows, available on all operating systems that support Windows Presentation Foundation (WPF).\u201d<\/i> UIA and WPF were developed hand in hand (since the first release of WPF). That is to say that the support for WPF UI elements through UIA clients is better than when using MSAA. To enable us to keep our automated UI regression tests, we had to re-write large parts, especially in the navigational portions of the framework that dealt with finding elements, to use UIA instead.<\/p>\n<p>Taking the previous examples, the UIA equivalent would look something like this:<\/p>\n<blockquote><p><b>Status Bar example:<\/b><\/p>\n<p>var mainWindow = AutomationElement.FromHandle(visualStudioHwnd);<\/p>\n<p>var statusBar = mainWindow.FindFirst(\nTreeScope.Children,\nnew PropertyCondition(AutomationElement.ControlTypeProperty, ControlType.StatusBar));<\/p>\n<p>\/\/ For VS2010 Text for status bar is now within the first child\nvar firstStatusBarCell = statusBar.FindFirst(TreeScope.Children, Condition.TrueCondition);\nstring statusBarText = firstStatusBarCell.Current.Name;<\/p><\/blockquote>\n<blockquote><p><b>Menu item example:<\/b><\/p>\n<p>\/\/ Get the main menu bar\nvar menuBar = mainWindow.FindFirst(TreeScope.Children,\nnew PropertyCondition(AutomationElement.ControlTypeProperty, ControlType.MenuBar));<\/p>\n<p>\/\/ Enumerate each child menu item\nforeach (AutomationElement child in menuBar.FindAll(\nTreeScope.Children,\nnew PropertyCondition(AutomationElement.ControlTypeProperty, ControlType.MenuItem)))\n{\n\/\/ Click it\nPoint clickable = child.GetClickablePoint();\nNativeMethods.SetCursorPos((int)clickable.X, (int)clickable.Y);\nNativeMethods.SendInput(\nNativeMethods.MOUSEINPUT.FromFlags(NativeMethods.MOUSEEVENTF_LEFTDOWN));\nNativeMethods.SendInput(\nNativeMethods.MOUSEINPUT.FromFlags(NativeMethods.MOUSEEVENTF_LEFTUP));\nThread.Sleep(100);\n}<\/p><\/blockquote>\n<p>It is also worth noting that UIA supports a concept called <a href=\"http:\/\/msdn.microsoft.com\/en-us\/library\/ms752362.aspx\">patterns<\/a>, which are, to quote MSDN, <i>\u201ca way to categorize and expose a control&#8217;s functionality independent of the control type or the appearance of the control.\u201d<\/i> IAccessible supported \u201cDoDefaultAction\u201d, whereas UIA supports many different types of actions. Taking the example above, instead of a mouse click one could get the \u201cExpandCollapsePattern\u201d and call the \u201cExpand\u201d method to expand each menu item.<\/p>\n<blockquote>\n<pre class=\"code\">foreach (AutomationElement child in ...)\r\n{\r\n    object pattern;\r\n    if (child.TryGetCurrentPattern(ExpandCollapsePattern.Pattern, out pattern))\r\n    {\r\n        ExpandCollapsePattern expandPattern = (ExpandCollapsePattern)pattern;\r\n        expandPattern.Expand();\r\n    }\r\n}<\/pre>\n<\/blockquote>\n<p>&nbsp;<\/p>\n<h3>Intermezzo #2: Updated numbers<\/h3>\n<ul>\n<li>Visual Studio 2010 has around <b>~70,000<\/b> automated UI regression tests. This includes tests for new features, new tests for old features.<\/li>\n<li>The core UI automation framework increased by ~<b>10,000<\/b> lines of code to ~<b>51,000 <\/b>lines of code. Much of this work was shimming the framework onto the same underlying technology used by the <a href=\"http:\/\/msdn.microsoft.com\/en-us\/library\/dd286726(VS.100).aspx\">Coded UI Test<\/a> feature that is shipping with Visual Studio 2010 (<a title=\"_GoBack\" name=\"_GoBack\"><\/a>which supports UIA).<\/li>\n<li>The core Visual Studio framework increased by ~<b>21,000<\/b> lines of code to ~<b>168,000<\/b> lines of code. Along with new features, this includes backwardly compatible automation support for the features that changed to use WPF for Visual Studio 2010. Most notably the windowing system, menu\/toolbar support and the editor.<\/li>\n<li>The other Visual Studio parts of the framework increased by ~<b>20,000<\/b> lines of code to ~<b>420,000 <\/b>lines of code \u2013 this most mostly new feature development.<\/li>\n<\/ul>\n<h3>Lessons from using UIA<\/h3>\n<p>In the process of migrating to the WPF shell we learned many lessons. I\u2019d like to share a few key ones in regards to automated UI testing.<\/p>\n<ul>\n<li>A \u201cfew\u201d tactical re-writes (most notably Tool Windows, Menus\/Toolbars and Editor) can go a long way. Having a framework that abstracted our UI was a life saver for shipping 2010. Small teams could iterate quickly on getting their automation support up to par using existing regressions tests as a meter of success.<\/li>\n<li>There is not really such a thing as an \u201cinvisible\u201d or \u201cbackground\u201d elements in UIA \u2013 when an element is gone, it will no longer be in the UIA tree. This is a good thing for assistive technologies (less pruning of the tree required), but can be troublesome for UI Automation frameworks. Historically we would tend to cache UI elements so we could get back to them again faster. When needed, one could call a mix of <a href=\"http:\/\/msdn.microsoft.com\/en-us\/library\/ms633548(VS.85).aspx\">ShowWindow<\/a> and <a href=\"http:\/\/msdn.microsoft.com\/en-us\/library\/dd318474(VS.85).aspx\">accSelect<\/a> with <a href=\"http:\/\/msdn.microsoft.com\/en-us\/library\/dd373634(VS.85).aspx\">SELFLAG_TAKEFOCUS<\/a>. This no longer works. If, say, a tool window is tabbed and not in the foreground; our cached element for it will because invalid, and if it\u2019s requested we need to a) bring it into the foreground and b) find the element from scratch again.<\/li>\n<li>HWND searching via various user32 API calls is fast. Using IAccessible is reasonably fast. UIA is not as fast. Net effect: running our tests became more expensive as they take more machine time.<\/li>\n<li>The inverse of the above is true in terms of fidelity; UIA gives the most attractive set of automation interfaces for test case and framework authors.<\/li>\n<li>UIA has the capability to proxy MSAA and UIA elements and UIA as MSAA elements. However, when the proxy is created some information may be lost in translation. Generally it\u2019s pretty good but there are edge cases.<\/li>\n<li>The majority of work in UI tests and UI test frameworks is spent finding the thing you want. You need a good story for this; after all a mouse click is always just a mouse click &#8211; it doesn\u2019t change. The same is true of sending keyboard input.<\/li>\n<li>The Win32 HWND class name of WPF\u2019s windows contains the name of AppDomain it was created in and a unique Guid. In other words \u201cwndclass_desked_gsk\u201d became, for example \u201cHwndWrapper[DefaultDomain;; c36a0d13-8c63-499f-b47c-5acecbdcac92]\u201d. Furthermore, the guid changes on every launch. This makes the main window difficult to search for. It turned out using the <a href=\"http:\/\/msdn.microsoft.com\/en-us\/library\/envdte(VS.80).aspx\">DTE<\/a>\u2019s <a href=\"http:\/\/msdn.microsoft.com\/en-us\/library\/envdte.window.hwnd(VS.80).aspx\">MainWindow.HWnd<\/a> property was the best way to get the main window handle for Visual Studio.<\/li>\n<\/ul>\n<h3>Tools<\/h3>\n<p>If you feel inspired to create automated UI regression tests for your product you may find the following set of tools useful.<\/p>\n<ul>\n<li><a href=\"http:\/\/msdn.microsoft.com\/en-us\/library\/dd286726(VS.100).aspx\">Coded UI Test<\/a> \u2013 New to Visual Studio 2010, a test type that enables record and playback of automation UI tests.<\/li>\n<li><a href=\"http:\/\/www.codeplex.com\/white\">White<\/a> \u2013 UI automation framework<\/li>\n<li><a href=\"http:\/\/www.codeplex.com\/UIAutomationVerify\">UIA Verify<\/a> \u2013 Verifies UIA pattern implementation for an application<\/li>\n<li><a href=\"http:\/\/msdn.microsoft.com\/en-us\/library\/ee872034(VS.85).aspx\">UI Spy<\/a> \u2013 Allows you to observe the UIA tree<\/li>\n<li><a href=\"http:\/\/www.microsoft.com\/downloads\/details.aspx?familyid=3755582A-A707-460A-BF21-1373316E13F0\">Active Accessibility 2.0 SDK Tools<\/a> \u2013 Contains various tools, one of which, AccExplorer, allows you to observe the IAccessible tree<\/li>\n<li>Spy++ (installs with Visual Studio) \u2013 allows you to observe the HWND tree<\/li>\n<\/ul>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/4\/2019\/06\/image_2.png\"><img decoding=\"async\" title=\"image\" src=\"https:\/\/devblogs.microsoft.com\/visualstudio\/wp-content\/uploads\/sites\/4\/2010\/03\/image_thumb.png\" alt=\"image\" width=\"112\" height=\"112\" align=\"left\" border=\"0\" \/><\/a><strong>Phil Price, <\/strong>Software Design Engineer in Test, Visual Studio Platform<\/p>\n<p><strong>Bio<\/strong>: Phil has been an SDET at Microsoft for 4 years. Previously working on the project system for Visual Studio 2008 and more recently on the WPF shell window management features.<\/p>\n<h3><\/h3>\n<h3><\/h3>\n<h3><\/h3>\n<h3>Thanks<\/h3>\n<p>A big thank you to Phil for his expert insights. Next time, <a href=\"https:\/\/devblogs.microsoft.com\/visualstudio\/wpf-in-visual-studio-2010-part-7-wrap-up\">Part 7 will conclude<\/a> the WPF in Visual Studio 2010 series with a short round-up of WPF 4 topics that we didn\u2019t have time to cover elsewhere <em>&#8211; Paul Harrington<\/em><\/p>\n<p>Previous posts in the series:<\/p>\n<ul>\n<li><a href=\"https:\/\/devblogs.microsoft.com\/visualstudio\/wpf-in-visual-studio-2010-part-1-introduction\/\">Part 1 : Introduction<\/a><\/li>\n<li><a href=\"https:\/\/devblogs.microsoft.com\/visualstudio\/wpf-in-visual-studio-2010-part-2-performance-tuning\/\">Part 2 : Performance<\/a><\/li>\n<li><a href=\"https:\/\/devblogs.microsoft.com\/visualstudio\/wpf-in-visual-studio-2010-part-3-focus-and-activation\/\">Part 3 : Focus and Activation<\/a><\/li>\n<li><a href=\"https:\/\/devblogs.microsoft.com\/visualstudio\/wpf-in-visual-studio-2010-part-4-direct-hosting-of-wpf-content\/\">Part 4 : Direct Hosting of WPF content<\/a><\/li>\n<li><a href=\"https:\/\/devblogs.microsoft.com\/visualstudio\/wpf-in-visual-studio-2010-part-5-window-management\/\">Part 5 : Window Management<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>This is the sixth article in the WPF in Visual Studio 2010 series. This week, guest author Phil Price gives us his view of what it took to test the new WPF UI. &#8211; Paul Harrington Background This post covers an overview of techniques that we used to create and maintain automated user interface regression [&hellip;]<\/p>\n","protected":false},"author":13,"featured_media":255385,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[155],"tags":[20],"class_list":["post-2243","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-visual-studio","tag-wpf"],"acf":[],"blog_post_summary":"<p>This is the sixth article in the WPF in Visual Studio 2010 series. This week, guest author Phil Price gives us his view of what it took to test the new WPF UI. &#8211; Paul Harrington Background This post covers an overview of techniques that we used to create and maintain automated user interface regression [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/visualstudio\/wp-json\/wp\/v2\/posts\/2243","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/visualstudio\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/visualstudio\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/visualstudio\/wp-json\/wp\/v2\/users\/13"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/visualstudio\/wp-json\/wp\/v2\/comments?post=2243"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/visualstudio\/wp-json\/wp\/v2\/posts\/2243\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/visualstudio\/wp-json\/wp\/v2\/media\/255385"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/visualstudio\/wp-json\/wp\/v2\/media?parent=2243"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/visualstudio\/wp-json\/wp\/v2\/categories?post=2243"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/visualstudio\/wp-json\/wp\/v2\/tags?post=2243"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}