FAQs for load testing

Article
12/07/2018

Azure Test Plans

Note

While Azure DevOps cloud-based load testing service is deprecated, Azure Load Testing Preview is available. Azure Load Testing Preview is a fully managed load testing service that enables you to use existing Apache JMeter scripts to generate high-scale load. To learn more, see What is Azure Load Testing Preview?. To learn more about the deprecation of Azure DevOps load testing and other, alternative services see Changes to load test functionality in Visual Studio and cloud load testing in Azure DevOps.

General

Go to related topic >

Pricing for Azure DevOps features

Q: How do I learn more about Cloud-based Load Testing?

A: Watch this video, or check out the Cloud-based Load Testing blog.

Q: Do I need anything to load test in the cloud with Visual Studio Ultimate 2013?

A: Yes, you'll need Update 4 or Update 5 installed. Download this version.

Q: My Visual Studio trial period ended, but I still want to run load tests.

A: To continue load testing after the trial, you'll need an active and valid Visual Studio Enterprise 2017 or 2015 license, or a Visual Studio Ultimate 2013 license. Learn more about licensing.

Q: Can I run cloud-based load tests on any app, even behind a firewall?

A: Yes, you can load test apps or sites that are only available to your company, such as internal or pre-release apps, staging or preproduction deployments. To learn more, see Testing private and intranet applications using cloud-based load testing.

Or, you can run a load test locally using Visual Studio.

Q: Can I run cloud-based load tests on my on-premises servers or on an Azure Virtual Machine?

A: Yes, several scenarios are supported. To learn more, see Run cloud-based load tests using your own machines.

Q: What are virtual users?

A: Virtual users create load by accessing your app or web site all at the same time during your test run. That way, you can test performance under more realistic or projected conditions. Virtual users are simulated by test agents.

Q: What are test agents? How do they relate to my test run?

A: Test agents are computing resources, like CPU, memory, and network, that generate load by simulating virtual users. Test agents use agent cores to create virtual users. Each core creates at least 1 virtual user.

For load test runs in Azure DevOps with the Visual Studio IDE, you can specify the number of cores to use. For example, if you get errors when you run your test, you might have to increase the number of cores.

Otherwise, your tests and the number of virtual users that you specify determine how many cores and agents are used.

Q: Where do I specify the number of cores for runs in Azure DevOps with the Visual Studio IDE?

A: You can do that here:

Update the agent count total cores

What do the values mean?

Cores	Agents
0	(Default) The number of cores is based on the number of virtual users that you specify for your test.
1	Your test run will use 1 agent.
2 - 10	Each agent will always use 2 cores.
11 - 40	Each agent will always use 4 cores.
41 - 200	Each agent will always use 8 cores.

The maximum number of cores for each test run is 200. If your test run needs more cores, you can run up to 10 load tests at the same time.

The minimum number of virtual users per agent core is 1. If your load test requires more cores, contact vsoloadtest@microsoft.com.

The number of agents also depends on your text mix (web performance test or unit test). If you have only web performance tests, we suggest using between 600 and 2,500 virtual users for every two cores. If you have unit tests, the agent count depends on what your unit tests do. This means you will have to test if you have enough agents by running a shorter duration load test run or use goal-based load testing.

Q: What are virtual user minutes (VUMs)? How many minutes will my load test use?

A: If your test run uses 25 or more virtual users per core, then VUMs = (max virtual user load for your test run) * (test run duration in minutes).

If your test run uses fewer than 25 users per core, then VUMs = (number of cores) * (25 virtual users per core) * (test run duration in minutes).

The minimum values used to calculate VUMs are 25 virtual users and 1 minute. If your test run values are smaller than the minimum values, then those values are rounded up to meet the minimums. For example, if your test run specifies 20 virtual users for 30 seconds, then your test run will actually run with 25 virtual users for 1 minute = 25 VUMs, not 15 VUMs.

Also, test run duration is in minutes, not seconds. For example, if your test run duration is 5 minutes and 15 seconds, then that duration is rounded up to 6 minutes.

A minimum of 250 virtual user minutes, including the warm-up period, is deducted from your subscription for:

Completed runs, based on the full duration of the run
Aborted runs, based on the elapsed run duration

For runs that end in an error state, no virtual user minutes will be deducted from your subscription.

Note that there is an additional charge for resource retention.

Pricing for Azure DevOps features

Q: I'm running load tests using test iterations. How is load test duration determined?

A: When load tests are configured to run using test iterations, the test duration cannot be determined when the test run starts. In such cases, the load test run assumes the maximum duration of 48 hours. If the test iterations complete in this time, the duration used for calculating the virtual user minutes consumed by your load test is the actual duration for which the test ran. If the test iterations do not complete in 48 hours, the load test will be stopped after 48 hours and virtual user minutes will be charged accordingly.

Q: Are there any limits when running the cloud-based load tests?

A: Yes. Based on where you're running the test, each test run duration limit is:

Visual Studio IDE: 48 hours
Azure DevOps Load test page:
- URL-based load tests: 48 hours
- JMeter load tests: 4 hours
Azure portal: 1 hour

Q: Do other load tests run on the same virtual machines that host my agents?

A: No, the virtual machines that host your agents host only one load test run.

Q: Are there load test features that aren't supported when you run load tests in the cloud?

A: These features aren't currently supported:

Network mix property
Agent to Use in test settings - use the core count property instead
SQL Trace properties in run settings
IP switching

Q: Can I speed up my load testing cycle by retaining the resources the tests use?

A: Yes, you can reduce the delay while your load test starts each time during a typical test > fix > retest scenario by retaining the resources for an appropriate period after each test completes, instead of having to wait for them to be acquired and provisioned for each test run. In Visual Studio Update 3 and later, specify the retention time in your run settings properties.

Specify resource retention time in Visual Studio 2017 and Visual Studio 2015 Update 3 and higher

In earlier versions of Visual Studio, add a context parameter named ResourcesRetentionTimeInMinutes to your run settings.

Specify resource retention time in Visual Studio 2015 pre Update 3

Note:

The maximum resource retention period is 4 hours (240 minutes).
The resources will be released after the specified period if you do not start another test during that period. When you start another test, the resource retention period you specify in that test will be applied - effectively allowing you to extend the retention period. You can specify a different retention period each time you start a test.
There a small additional charge applied when you use this feature. In addition to the usual VUMs that your load test consumes, a surcharge of 5 VUMs per core is applied for the duration of the run. So if you retain 20 cores for 10 minutes, an additional 1000 VUMs (5 x 20 x 10) will be charged. The status messages display this information.
Resource retention is not available for Apache JMeter tests at the present time.

For more details, see this blog post.

Q: How do I delete a load test?

A: Currently, only test runs can be deleted, and only by the user that created that test run. The load test itself cannot be deleted because the data resides at the subscription level.

Visual Studio load testing

Go to related topic >

Q: How can I increase the capacity of my load tests?

A: You can use the Cloud-based Load Testing service, so you can run your tests across multiple virtual machines in the cloud.

Q: How many virtual users can I configure in my load test?

A: In the full version of Visual Studio Enterprise, the number of virtual users is unlimited. In Visual Studio Enterprise trial version, the virtual user count is limited to 250.

Q: Can I analyze load tests that ran previously?

A: Yes, to open and manage those results, click Manage results toolbar button in the load test editor. You can have multiple tests open at the same time to compare runs, and create trend analysis reports to compare them.

Q: Is there a difference between what I can analyze during a running test versus a completed test?

A: Yes, these are the differences:

Performance counters A smaller subset of the performance counter data is available while a test is running.
Views When the load test run has completed, the Summary View and Details View are available.

Q: Can load tests use other test types in their test mix besides web performance tests?

A: Yes, you can include unit tests and coded UI tests.

Q: Can virtual users simulate pausing between test steps?

A: Yes, you can specify think times to simulate the time spent by a user on a web page.

Q: Why should I use Cloud-based Load Testing?

A: If you don't want to set up machines for load testing, or you don't have available resources, you can use the Cloud-based Load Testing service. It sets up virtual machines in the cloud that will run your load test. Note that your web site must be publicly available on the internet for load testing using Azure DevOps to access it.

Setting up tests

Go to related topic >

Q: Can I have other test types, besides web performance tests, in a load test mix?

A: Yes, you can include unit tests and coded web tests, but not coded UI tests.

Q: How long do I have to wait until I can run my load test after creating an Azure DevOps subscription?

A: It can take between 5 seconds to 3 hours until you get permissions to run the load test in the cloud. If you previously created your Azure DevOps subscription, you might be able to run the load test right away.

Q: How do I provide different values to the same test?

A: Use a .csv file or an Excel spreadsheet. Using SQL Server is currently not supported. Learn how to supply values to your test.

Q: Help, I'm having problems with my agents!

A: You must have at least 1 virtual user per core. If you're getting status messages that an agent stopped working due to load, or if the downloaded report shows high CPU use for an agent, try increasing the number of agents that you're using.

If you need more help, contact vsoloadtest@microsoft.com

Q: Where are the test agents used for my load test runs located?

A: When you set up your load test run, you can select the test agent location from any supported Azure datacenter, starting with Visual Studio Ultimate 2013 Update 5 and Visual Studio Enterprise 2015. After your run finishes, your results are stored in the same location as your Azure DevOps subscription.

Edit load test to set location

If you're using an earlier version of Visual Studio, the agent location is based on the location that you chose when you created your Azure DevOps subscription.

Azure DevOps Region	Test Agent Azure Datacenter
South Central US	East US 2
West Europe	West Europe

Q: Can virtual users simulate pausing between test steps?

A: Yes, you can specify think times. Select a scenario in your load test and edit the think time in the Properties view.

Q: Where can I get more information about simulating real-world loads?

A: Learn more about how to specify web performance test properties, load test scenario properties, and run settings properties.

Q: Can I run load tests locally and in the cloud from the same project?

A: Yes, your project can have multiple test settings files. Add another test settings file to your Solution Items folder.

Right-click test settings file, click Active Load and Web Test Settings

Now you can use one settings file to run your tests locally and the other settings file to run your load tests in the cloud. To switch between them, open the file's shortcut menu, then select the test settings file that you want to use.

Q: How do I install certificates or software on agents that run my load tests in the cloud?

A: In the test settings, you can use deployment options and a setup script. In the Deployment window, add the .exe or other files that you want to deploy on the agents. To install those files on the agents, use the setup script.

All the items deployed on the agents are copied to a directory on the agent. You can access the directory location by using %DeploymentDirectory% in the setup and cleanup script. For example, if you want to install WebDeploy on the agent machine, add WebDeploy_x64_en-US.msi to Deployment window. The setup.cmd will look like this:

%DeploymentDirectory%\WebDeploy_x64_en-US.msi /passive

Apache JMeter tests

Go to related topic >

Q: What are the supported JMeter versions?

A: The latest supported version of Apache JMeter is version 3.2.

Q: Which samplers are supported?

A: Currently, only HTTP samplers are supported.

Q: I want to supply properties to JMeter. How do I do that?

A: Add the properties you need to a user.properties file. Upload it using the Supporting files parameter when you set up the test, and it will be applied when the load test runs.

Q: Are custom listeners supported?

A: Custom listeners are not currently supported.

Running and monitoring load tests

Go to related topic >

Q: Can I use mstest to run load tests with Azure DevOps?

A: Yes, you can in Azure DevOps, and in TFS 2015 and later. For more information, see this blog post.

Q: Can I debug a load test while it's running in the cloud?

A: Yes, you can do this when you use Visual Studio Enterprise 2015 or later. Learn more.

Q: How can I check the status of the Cloud-based Load Testing service?

A: You can view the service status at the top of the Azure DevOps support page and on our service blog. You can also subscribe to alerts for service status by following this post in our support forum.

Q: What are the possible load test run states?

A: When you run load tests with Azure DevOps, the test run states are:

In-Progress: The test run is currently running in the cloud.
Completed: The test run was completed successfully.
Aborted: The test run was stopped because the user clicked the stop button. This state can also result from issues related to your load test, such as issues with your test scripts.
Error: The test run was stopped due to an error with the service itself. For example, there might be an infrastructure issue in the service, and it can't continue to run your test. This is not an issue caused by your load test or test scripts.

Q: Where is my load test report stored after I download it?

A: Your downloaded reports are stored in a local SQL Server Express database. You can change the default location, if you want. You can also store all the reports together for everyone by changing the location for each user to the same database.

SQL Server Express works best for storing test results from a trial run. For better performance as you download more reports, use SQL Server.

Q: How should I view test logs after downloading the test results locally?

A: Due to a known issue, you must currently use this workaround:

Start Notepad with administrator privileges.
Open devenv.exe.config file. You can usually find this file at: C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE

Change the value of bindingRedirect to "8.0.0.0-14.0.0.0"

<dependentAssembly>
  <assemblyIdentity name="Microsoft.VisualStudio.QualityTools.LoadTest" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
  <bindingRedirect oldVersion="8.0.0.0-14.0.0.0" newVersion="12.0.0.0"/>
</dependentAssembly>

Recording and replaying tests

Go to related topic >

Q: Can I simulate actions from different users and data-drive my test?

A: No, this capability is planned but is not available at present.

Q: Does the feature automatically identify and correlate all dynamic information from all requests?

A: No, this capability is currently supported only for the ASP.NET information such as VIEWSTATE, EVENTVALIDATION, and similar values. Your request may fail if it contains other dynamic information. In these cases, you should run the tests using Visual Studio.

Q: Can I test REST APIs using the functionality provided by the Azure DevOps portal?

A: Yes, you can use the URL-based test to test REST APIs. Enter the request URL of your API and the details required to create your test.

Q: I want to see what's possible in Visual Studio after I export the test. Do I need to have Visual Studio Enterprise edition?

A: No, you can use the 90-day trial version of Visual Studio Enterprise edition. This allows you to run cloud-based load tests. See this blog post.

Application Insights

Go to related topic >

Q: Can I define load tests once and then run them against different hosts?

A: Yes. Use context parameters to pass information and change values for parameters in the load test:

Parameterize the URLs in your web test
Add matching context parameters to your load test runsettings file. To do this expand Run Settings in the load test editor, right-click your [Active] run settings entry, then choose Add Context Parameter.

If you are running load tests in the CI/CD pipeline where the URL of the app may change from stage to stage, you can use the Cloud Load Test task. It enables you to override the context parameter value from the task UI and use any stage variables defined in the pipeline to pass in the required information.

Q: Can I get more detailed profiler information?

A: Yes, see Profiling live Azure web apps with Application Insights.

Q: Can I view data from other app monitoring tools when load testing in the cloud?

A: No.

Q: Can I increase how often data is collected?

A: No, this is currently a fixed frequency of one minute.

Q: I don't see any counters even after waiting a few minutes. What's wrong?

A: Go to Application Insights and check that you can view performance data for your app there. If you see data collected there, report your issue to vsoloadtest@microsoft.com.

A: This can happen because:

No apps are configured to push analytics data to Application Insights. See Get started with Visual Studio Application Insights. Also check that you can see the apps in Application Insights in the Azure portal, as shown here:
The Azure Resource Manager access token has expired. The token is valid for 12 hours in the context of Azure DevOps. Sign out of Azure DevOps and then sign in again to refresh the token.
Azure Active Directory is not enabled for your Azure DevOps subscription. See Access Azure DevOps with Azure Active Directory.

If none of the above works, contact us at vsoloadtest@microsoft.com.

Troubleshooting

Q: What do I do if Visual Studio stops responding when I try run a load test in the cloud?

A: To resolve this issue, see Known issues with load testing.

Q: How do I record a web performance test with Internet Explorer 11?

A: If the web test recorder is not active when you try and record your web test with Internet Explorer 11, see Using Internet Explorer 11 and not able to record a web performance test to resolve the issue.

Q: How do I view errors and warnings that happen when my load test is running in the cloud?

A: Status messages and test errors are reported while your load test runs. Status messages give you details about the load test run itself, such as when a connection to the results database is lost. Test errors relate to the test. View both these messages from the Details page on the progress graphs.

View status and error messages

Q: I get an error when I try to import downloaded test results. What do I do?

A: If the error states that the connection's current state is closed, you can set the amount of time that a connection waits before timing out.

Set the ConnectTimeout or Connection Timeout keywords in the connection string. Do not set a value of 0 as a timeout in a ConnectionString because the connection will keep trying to connect indefinitely.

Q: Why can't I use more than 250 virtual users or plug-ins when I have Visual Studio Ultimate or Visual Studio Enterprise?

A: If this happens, you must take the Visual Studio product key from your MSDN subscription and use the "Change my Product License" option on the Product Information page. You must do this on every machine where you want to run load tests using Azure DevOps. To get the product key, visit this site.

Q: Why did the REST API calls that I use stop working?

A: Starting on 26th November 2014, you must add the version information to your REST API calls. If your call fails with a VssVersionNotSpecifiedException exception, you must include ?api-version=1.0-preview.1 in your REST API calls. To do this, see Get started with the REST APIs.

Q: I noticed that user code fails to execute if it depends on the test names. Are test names changed when run against the service?

A: When the test runs using Azure DevOps, test names in load tests are converted to lower case. Any string match done on a test name by user code should ignore the case or convert test names to lower case.

Q: How do I enable client-side logs to help troubleshoot issues with load tests run in the cloud?

A: Edit devenv.exe.config with a text editor. You can usually find file at:

"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE"

Add this line inside the <appSettings< section:
```
<add key="ElsClientLogLevel" value="XXX"/>
```
Where XXX can be any of the following:
- all - logs all messages
- off - stops logging any messages
- critical - only logs critical messages
- error - only logs error and critical messages
- warning - logs error, critical and warning messages (default)
- information - logs error, critical, warning and info messages
- verbose - logs error, critical, warning, info and verbose messages

Add this section to the bottom of the devenv.exe.config file, just above the closing tag. You can specify the path for the log file by changing the initializeData value.

<system.diagnostics>
  <trace autoflush="true" indentsize="4">
   <listeners>
    <add name="myListener" type="System.Diagnostics.TextWriterTraceListener" initializeData="d:\VSTestHost.log"/>
   </listeners>
  </trace>
  <switches>
   <!-- You must use integral values for "value": 0 = off, 1 = error, 2 = warn, 3 = info, 4 = verbose.-->
   <add name="EqtTraceLevel" value="4" />
  </switches>
</system.diagnostics>

Restart Visual Studio and reproduce the issue. You can then review the log file or share it with Support. You can find the log file at %Temp%\ELSClient\.

Q: Why don't I see the individual timing values in the Load Tests Results Store?

A: For Visual Studio 2013 Update 4, Visual Studio Enterprise 2015, and later versions, the default value for the TimingDetailsStorage property was changed from AllIndividualDetails to None. If you want to collect the individual timings, you must specifically set TimingDetailsStorage property to be AllIndividualDetails. See Load Test Run Settings Properties.

Errors

Q: My test run failed with these errors. What do I do?

A: If you get one of these errors:

VS1550064
VS1550072
VS1550078
VS1550081
VS1550082
VS1550083

Contact Azure DevOps support. You will have to give them your test run ID.

Q: My run was aborted because the .loadtest xml file could not be parsed. What do I do?

A: You might get these errors if you manually edit the .loadtest xml file:

VS1550084

Open the file and revert any changes that you added. Rerun the load test. The run should complete successfully.

Q: Too many applications or counters were selected to run for my load test. What do I do?

A: You might get these errors if you manually edit the .loadtest xml file:

VS1550026
VS1550027

Open the file and revert any changes that you added. Rerun the load test. The run should complete successfully.

Q: No active load test settings were found in my load test. What do I do?

A: You might get this error if you close the load test wizard without completing it:

VS1550030

To fix this problem, create another load test. Delete the failed test run.

Q: My load test got an error when it started or was aborted during the run. What do I do?

A: Generally, these problems happen due to issues with the cloud-based load testing service. Just try and run your load test again. If these problems still happen, contact Azure DevOps support. You will have to give them your test run ID.

Q: Where can I find information about other errors?

A: See Visual Studio Cloud Load Testing error codes to find more details about other errors and their resolutions, where applicable.

Links to useful resources

Tutorials

Case Studies

Blogs and other references

Samples

Help and support

Report any problems on Developer Community, get advice on Stack Overflow, and get support via our Support page.

Share via

FAQs for load testing

General

Q: How do I learn more about Cloud-based Load Testing?

Q: Do I need anything to load test in the cloud with Visual Studio Ultimate 2013?

Q: My Visual Studio trial period ended, but I still want to run load tests.

Q: Can I run cloud-based load tests on any app, even behind a firewall?

Q: Can I run cloud-based load tests on my on-premises servers or on an Azure Virtual Machine?

Q: What are virtual users?

Q: What are test agents? How do they relate to my test run?

Q: Where do I specify the number of cores for runs in Azure DevOps with the Visual Studio IDE?

Q: What are virtual user minutes (VUMs)? How many minutes will my load test use?

Q: I'm running load tests using test iterations. How is load test duration determined?

Q: Are there any limits when running the cloud-based load tests?

Q: Do other load tests run on the same virtual machines that host my agents?

Q: Are there load test features that aren't supported when you run load tests in the cloud?

Q: Can I speed up my load testing cycle by retaining the resources the tests use?

Q: How do I delete a load test?

Visual Studio load testing

Q: How can I increase the capacity of my load tests?

Q: How many virtual users can I configure in my load test?

Q: Can I analyze load tests that ran previously?

Q: Is there a difference between what I can analyze during a running test versus a completed test?

Q: Can load tests use other test types in their test mix besides web performance tests?

Q: Can virtual users simulate pausing between test steps?

Q: Why should I use Cloud-based Load Testing?

Setting up tests

Q: Can I have other test types, besides web performance tests, in a load test mix?

Q: How long do I have to wait until I can run my load test after creating an Azure DevOps subscription?

Q: How do I provide different values to the same test?

Q: Help, I'm having problems with my agents!

Q: Where are the test agents used for my load test runs located?

Q: Can virtual users simulate pausing between test steps?

Q: Where can I get more information about simulating real-world loads?

Q: Can I run load tests locally and in the cloud from the same project?

Q: How do I install certificates or software on agents that run my load tests in the cloud?

Apache JMeter tests

Q: What are the supported JMeter versions?

Q: Which samplers are supported?

Q: I want to supply properties to JMeter. How do I do that?

Q: Are custom listeners supported?

Running and monitoring load tests

Q: Can I use mstest to run load tests with Azure DevOps?

Q: Can I debug a load test while it's running in the cloud?

Q: How can I check the status of the Cloud-based Load Testing service?

Q: What are the possible load test run states?

Q: Where is my load test report stored after I download it?

Q: How should I view test logs after downloading the test results locally?

Recording and replaying tests

Q: Can I simulate actions from different users and data-drive my test?

Q: Does the feature automatically identify and correlate all dynamic information from all requests?

Q: Can I test REST APIs using the functionality provided by the Azure DevOps portal?

Q: I want to see what's possible in Visual Studio after I export the test. Do I need to have Visual Studio Enterprise edition?

Application Insights

Q: Can I define load tests once and then run them against different hosts?

Q: Can I get more detailed profiler information?

Q: Can I view data from other app monitoring tools when load testing in the cloud?

Q: Can I increase how often data is collected?

Q: I don't see any counters even after waiting a few minutes. What's wrong?

Q: Why do I get an "Unable to connect to Azure DevOps due to network failure" error when trying to add apps using the Get Performance Data from Application Insights menu command?

Troubleshooting

Q: What do I do if Visual Studio stops responding when I try run a load test in the cloud?

Q: How do I record a web performance test with Internet Explorer 11?

Q: How do I view errors and warnings that happen when my load test is running in the cloud?

Q: I get an error when I try to import downloaded test results. What do I do?

Q: Why can't I use more than 250 virtual users or plug-ins when I have Visual Studio Ultimate or Visual Studio Enterprise?

Q: Why did the REST API calls that I use stop working?

Q: I noticed that user code fails to execute if it depends on the test names. Are test names changed when run against the service?

Q: How do I enable client-side logs to help troubleshoot issues with load tests run in the cloud?

Q: Why don't I see the individual timing values in the Load Tests Results Store?

Errors

Q: My test run failed with these errors. What do I do?

Q: My run was aborted because the .loadtest xml file could not be parsed. What do I do?

Q: Too many applications or counters were selected to run for my load test. What do I do?

Q: No active load test settings were found in my load test. What do I do?

Q: My load test got an error when it started or was aborted during the run. What do I do?

Q: Where can I find information about other errors?

Links to useful resources

Help and support

Additional resources