Visual Studio 11 Beta: Manual Testing of Windows Metro Style Apps
NOTE: This article assumes that you are familiar with Microsoft Test Manager. See this *MSDN article for a full description of Manual Testing features available with Visual Studio 11. This article focuses on the features that are specific to Windows Metro style apps available in Windows Consumer Preview.*
**
With Dev11 Beta, now available at this download location, you’ll find that Microsoft Test Manager has been enhanced to enable testing of Windows Metro style apps on a device. At this point in Dev11 development, you’ll see that the HTML/JavaScript/CSS scenarios work pretty well; there are some known issues in XAML scenarios (mainly impacting quality of the text action log), although you’ll find that the image action log works well for all Windows Metro style apps. Please give it a try and give us your feedback! And for a full walkthrough of this experience, look at the following MSDN article: Testing Windows Metro Style Apps . This blog highlights the new features to enable this experience.
Testing on a remote device
You can now specify a remote device for testing.
The following data diagnostic adapters are available for Windows 8 Devices and are turned on by default.
1. Event Log
2. System info and
3. Action log
IntelliTrace and Test Impact Data Diagnostic Adapters are not yet supported on Windows Metro style apps. If you want to collect additional information during testing, you can install custom data diagnostic adapters [LINK TO INSTRUCTIONS] on the Windows device and then use them in test settings.
You can install a Windows metro style app from Microsoft Test Manager on a remote device.
If you have a Team Build assigned to your Test Plan, the file picker will default to the drops folder for that build. This will help the Manual Tester to quickly install the latest version of her Metro style app on the device.
Microsoft Test Manager takes care of installing Developer License, Certificate & the app itself.
Image Action Log
When you create a bug from Microsoft Test Manager, it now includes an Image Action Log. This is how the new Rich Bug looks like:
When you choose the ‘Action Log’ link, you get the Image Action log shown below.
Notice how the action log consists of a text description, a thumbnail of the control interacted with and a full screenshot at the time of action. You can hover over individual thumbnails to get the full screen image for that action.
As a rule, the thumbnail and the full screen image are taken the moment after the action is completed. What we have found is that, this “After Image” gives the right information for the developer who is attempting to repro the bug. There are a few exceptions to the rule. E.g:- When you click a hyperlink which navigates to a new page, we take a “Before Image”, because the “After Image” in most cases turned out to be the page navigation animation. We also show the exact interaction point on the control.
Image Action log is uploaded to the Team Foundation Server. We only upload this when you create a bug. This helps in reducing the space requirements on the Team Foundation Server.
Action Log Quality
The key quality metric for Action Log is how successful the developer is in repro’ing the bug based on the action log. To achieve this, we focus on two characteristics – Completeness & Readability.
Completeness requires that all actions performed by the user are captured. The following features help here.
1. Behind the scenes, we have attached low level hooks to Mouse & Keyboard.
2. We have built a Touch Input Redirect which listens to all touch input. We accurately record multi-touch gestures such as Rotate, Zoom & Pan too.
3. We use UI Automation to listen to events and identify controls.
4. We also dynamically generate events when we detect controls which are not firing appropriate events.
With this infrastructure, we are able to ensure a high level of completeness in the action log.
Readability requires that we have the correct name for each control in the action log. The following features help here.
1. We use UI Automation to retrieve the name of the control. UI Automation has been significantly improved to work with ARIA. ARIA allows a very easy way to specify names, events & roles in Html.
2. We use a smart vicinity algorithm to generate a name for the control based on the labels in its vicinity. We leverage established UI Patterns and UI guidelines to identify the appropriate labels for controls. Vicinity is applied for all UI Controls except radio button and checkbox.
Exploratory Testing
You can perform Exploratory Testing also on a remote device. Both the remote device selection & app installation experience are available during Exploratory Testing.
While doing Exploratory Testing on a remote device, you can see image thumbnails in your bug. You can also see the Images in the Change Steps dialog. In the Change Steps dialog, you can hover over the thumbnail to see the full screen image.
Before You Run
If you’re interested in using manual testing, you’ll want to think about a couple of things. First, for all the features described in this article, you need to install Microsoft Test Professional on a desktop machine. You also need to install Visual Studio Remote Tools on the Device.
In general, you’ll find that manual testing works better with two machines, but you can do it with just one (preferably one machine with two monitors). If you want to do that, just:
1. Install Visual Studio Remote Tools on your Windows 8 desktop where you have installed Microsoft Test Professional.
2. Start Microsoft Test Tools Adapter Configuration tool and start the service.
3. In the “Connect to Remote Device” dialog, specify the name of the desktop machine.
Send Your Feedback
We hope you like the new manual testing features for Metro style applications. Please send us your feedback on Connect or in the Forums. If you run into problems you can even turn on tracing, and ask questions on the Forums.
Light
Dark
0 comments