Testing Windows Store apps Running on a Device Using the Exploratory Test Window

You can do exploratory testing on your Windows Store apps that are running on a Windows 8 remote device such as a Windows 8 tablet.

Requirements

• Visual Studio Ultimate, Visual Studio Premium, Visual Studio Test Professional

Why should I do this?

By connecting to your Windows 8 remote device to test your Windows Store app, you can identify problems that users will run into when they use their devices in real-world situations. The problems that are typically identified when you run exploratory tests include incorrect functionality or bugs, poor performance, and security and connectivity issues. The outcome, resolutions, and comments that you can enter during an exploratory test are saved as test results. To test your other apps, you do not need to connect to a Windows 8 remote device. You can test the application on the same machine running Microsoft Test Manager. For more information about performing exploratory testing applications that are not Windows Store apps, see the topic: Performing Exploratory Testing Using Microsoft Test Manager.

How do I do this?

Step 1: Prepare the Windows 8 remote device for testing

Before you can run exploratory tests on your Windows 8 remote device, you must install the Remote Debugger on the device that you want to test. The Remote Debugger includes the Microsoft Test Tools Adapter, which is automatically installed and runs as a service on the remote device to enable testing. The remote debugger setup programs are available on the Visual Studio installation media in subfolders of the vs\Remote Debugger folder. If you want to see more information about the remote debugger before you try to install it on your Windows 8 remote device, see the topic: Installing the Remote Debugger.

Step 2: Connect to your remote Windows 8 device from Microsoft Test Manager

In this step, use the following illustrations as a guide for the steps that are required in order to connect to your remote Windows 8 device.

Connect to your remote Windows 8 device:

You will need to enter your credentials when prompted:

Step 3: Install you Windows Store app onto the remote device

Now that you’re connected to the remote device, you can install your Windows Store app onto your remote device for testing. Follow the steps in the illustrations below.

Installing your Windows Store app on the remote device:

Install your Windows developer license when prompted:

Install a testing certificate when prompted:

Step 4: Test your Windows Store app!

You should now be able to test your Windows Store app on your remote Windows 8 device. When you run exploratory tests on a Windows 8 remote device, you perform the actual exploratory testing on the remote device and add comments, screenshots, file attachments on the machine running the Exploratory Test window.

Now, test your Windows Store app:

Capturing issues during an exploratory test session

While you implement your exploratory test session, you can add notes, screenshots, and any applicable file attachments as you discover errors, requirements violations, or other discrepancies.

After you add your comments, screenshots, and file attachments, you can create a new bug. The new bug will automatically include these items. When you create a new bug, you can specify which action steps to include in the bug. For more information, see Exploring the application in the topic Performing Exploratory Testing Using Microsoft Test Manager.

You can also create a new manual test case that is based on the action steps that you have performed against the application you are exploring. Just like the bugs, the new manual test cases are listed within your session and can also be opened and edited as you continue to explore the application. For more information, see Make re-testing easy in the topic Performing Exploratory Testing Using Microsoft Test Manager.

Step 5: Submitting a bug

When you submit a bug, fill out the fields and save the bug. When you submit bugs for your Windows Store apps, you can select the specific steps to include in your bug and you can remove steps that are not necessary. The next illustration shows how to select the steps to include in the bug.

Selecting the steps to include in the bug

What else should I know?

Prerequisites

• Make sure you have the required version of Visual Studio installed. To use the exploratory testing window from Microsoft Test Manager, you must have Microsoft Visual Studio 2012 Testing Essentials, Premium or Ultimate installed.

• The remote Windows 8 device and the machine running Microsoft Test Manager must be connected over a network or connected directly through an Ethernet cable. Testing over the internet is not supported.

• To display the Microsoft Test Manager window, choose Start, and then choose All Programs. Point to Microsoft Visual Studio 2012 and then choose Microsoft Test Manager.

• When you start Microsoft Test Manager for the first time, you are not connected to any team project. You must first connect to Team Foundation Server, select your project collection, and your team project. The team project is the same one that is used by other users on your team to create requirements or user stories, maintain source code, and build the apps that you want to test. Your testing artifacts are created and managed in this same team project. For more information about connecting to a team project, see How to: Connect to a Team Project For the First Time from Microsoft Test Manager.

Important issues

• Security warning: After you have started an exploratory test session, the action steps that you perform on the application being explored are recorded automatically. Make sure that you protect any sensitive data. For example, avoid any unnecessary sensitive content in your screenshots and recordings.

• Security warning: You can optionally choose to run the Microsoft Test Tools Adapter (remote debugger) in No Authentication mode, but this mode is strongly discouraged. There is no network security when you run in this mode. Choose the No Authentication mode only if you are sure that the network is not at risk of from malicious or hostile traffic.

• Warning: Remember, when you run exploratory tests on a Windows 8 remote device, the test steps must be executed on the remote device. The test steps are marked as passed or failed on the machine runs Test Runner.

• Caution: Both the machine that is running Microsoft Test Manager for Visual Studio 2012 and the Windows 8 remote device should be running as the same user. If the user is different, a credentials dialog box will display when you try to connect. The dialog box will require you to provide the same credentials to connect to the remote device.

  You can also log on using the same Microsoft Live ID on both machines if your logon credentials differ.

• Caution: Testing Windows Store apps on Windows 8 remote devices that use the ARM architecture is not supported in Visual Studio 2012.

• Caution: To record audio, you must have an audio recording device configured on your computer.

Tips

• To connect directly to a remote device, connect the two machines with a standard Ethernet cable. If the device does not have an Ethernet port, you can use a USB to Ethernet adapter to connect to the cable.

• When testing a Windows Store app, an enhanced html action log is created. You can view it in your web browser. The action log includes screenshots that help identify the UI elements being used when a problem is found.

• By default, the port 6905 is used by Microsoft Test Manager to communicate with remote devices. You can override this port if necessary. If you use a different port, you must enter the IP address in the Device name/IP address field. For example, enter testDevice1:nnnn.

  You can view or change the port being used by the Microsoft Test Tools Adapter by opening the service configuration file mttaservice.exe.config in the Visual Studio installation folder, on the remote device.

• If something else requires your attention while the exploratory test session is running, you can pause the recording and use your computer for other activities. Those activities won’t be included in the action step recording. For example, you can switch to your e-mail application to respond to something critical and nothing will be recorded.

What else should I read?