Web and Load Testing with Visual Studio Team System

Continuing the series on Testing with Visual Studio Team System, I devote this blog to our Web and Load Test functionality.

The web test and load test features were initially introduced in Visual Studio 2005, and enhanced in Visual Studio 2008. These tests are available in the Team Test and Team Suite editions of Visual Studio Team System.

Web tests enable you to generate http requests and responses, test for correctness of the responses, and measure response times and throughput. The primary scenario for web tests is to use them in a load test to generate load against a web application and measure web application performance. In order to efficiently generate load, Web tests work at the http layer, they do not drive the browser.

Load tests enable you to simulate many users hitting using an application at the same time, and then measure server response times, throughput, error rates, and resource utilization on the servers under test. Load tests will help you ensure your applications will stand up under load and deliver acceptable performance. In addition to driving web tests to simulate user activity, load tests can also run “unit” tests under load, which enables you to drive load to any server with a .NET API.

Enterprise Ready

This feature set delivers powerful enterprise-class load and performance testing features to the Team System product line. In the years since it was developed it has become the de facto load testing solution used internally by Microsoft. Nearly every server product team and web property now uses Visual Studio load tests. Server teams using this feature set include SQL, Windows, VSTS, and SharePoint, CRM, Dynamics, Media Center, and many more. Web properties include Hot Mail, Live Earth, Ad Center, Cars, Music, and many more. It is also the de facto tool used by our IT teams, who test everything from accounts payable, to the payroll system, to our polling system using VSTS.

It has also been used by a wide variety of customers and load test consultants to test their mission critical systems, with scale to 100,000s of users, on systems as varied as mission critical customer applications, to huge SharePoint deployments, to delivery of television content over phone lines.

Introduction to Web Tests

As I stated above, the primary use of Web tests is to use them in a load test for performance testing.

In order to efficiently drive load, Web tests work at the http layer. So rather than driving IE or another browser, they generate http requests and responses. A key thing web tests do not do is execute java script. Any http requests sent in java script will be recorded and played back, but again by sending http requests, not by running the java script. Under the covers the Web test engine uses the system.net classes in the .NET framework for generating http traffic.

Web tests can also be used as functional test to validate the application server responses, but since they run at the http layer cannot test anything that happens in the browser: java script execution, page layout, or control behaviors.

You start by creating a test project, then adding a web test to the project:

Which launches the recorder:

Which generates a Web test in Visual Studio:

Which you can then playback:

Playback allows you to view response time, error code status, and response size. For a given request/response, you see a preview of the html that was returned, and you can view the actual request and response data. The context parameters shows what values were in the virtual users context, and you can also see the results of any validation or extraction rules.

Web tests do a lot under the covers to do this effectively. When Web tests are run the Web test engine automatically does many of the same things the browser does:

Handle cookies
The web test engine automatically tracks cookies for a given "virtual user", storing cookies in a cookie container when received from the server, then sending them back as required on subsequent requests.

AuthenticationNTLM, Kerberos, and forms authentication are all supported.

HTTPS Support
The Web test engine supports both http and https requests.

Automatically Download Resources Most web pages contain images, style sheets, and java script. The Web test engine will automatically parse html responses and fetch the resources on the page (also called "dependent requests").

Follow Redirects
By default the Web test engine automatically follows page redirects. 

Hidden Field TrackingThe web test recorder automatically generates code to extract hidden field values and bind them to subsequent requests.

Validation RulesA validation rule is used to pass or fail a test. There are a set of canned rules that ship with web tests, such as the validation text rule. This rule will fail if the specified text is not found on the page.

Extraction RulesAn extraction rule is used to scrape data from a response for use in a later request. A good example of this is the hidden field extraction rule which extracts the values of all hidden fields on a page for use in the subsequent post request.

Dynamic Parameter HandlingDynamic parameters are any parameters that change each time you run hit a web site. A great example is a session id, which is unique each time you create a new session with the web server. You can't simply replay the recorded session id, or the test will fail. Dynamic parameters are often stored in cookies and hidden fields, which are automatically handled as outlined above. Some querystring parameters also contain dynamic parameters, and the dynamic parameter detection tool detects these and automatically correlates them using extraction rules.

Data Driven TestsFor multi-user testing, you typically do not want each user entering the exact same data in the test. Data driven tests provide the ability to bind parameters to data from a data source. This enables you to easily test different data combinations.

ParameterizationYou can easily parameterize a web test. A great example of this is using the same test to target different web servers, which you can do using the Parameterize Web Server command. The creates a context parameter with a default value, which can then be overridden from a load test or from an environment variable.

Extensibility The Web test engine fires events to code before and after the test, and before and after each request, page, and transaction. This provides you with powerful hooks to a wide variety of things in your test. In addition, you can add your own extraction and validation rules.

"Declarative" or Coded Tests When you create a new web test, a "declarative" style test is generated. Declarative tests are persisted as xml and have a designer. You can also generate a coded web test, which enables you to do looping, conditional execution, and compute parameter values in code. Generally the designer is easier to work with.

Included TestWeb tests can also be broken up into parts, and the parts can be re-used to build other tests. This can make test maintenance easier, since if one part of the app breaks, you only have to fix the web test components that hit that part.

Introduction to Load Tests

Load tests enable to you simulate many users hitting an application at the same time. The primary scenario for load testing is performance testing. How will the server respond to so many users? Will response times be unacceptably slow? Will error rates be acceptable? Another is capacity planning, where you may want to understand how large a server you will need to support the expected user load.

You start out creating a load test by walking through the load test wizard. The wizard will guide you through creating a scenario, which you can think of as a particular group of users. The load test scenario primarily contains:

• the load pattern, which defines how users will running at any given point in the test
• the test mix, which defines which test scripts they will be using. A test should roughly be viewed as a single users walk through the app.

In addition to web tests, load tests can drive "unit" tests, which enables you to drive load to any server that can be called from managed code (which is nearly every server or protocol on the planet).

Counter sets allow you to easily control which counters are collected from the load agents and the system under test. It is very easy to configure the counters to collect on a given machine. Individual counters also contain thresholds, which when triggered fire special alerts in the load test monitor.

The run settings allow you to map which machines to collect counters from, as well as various other properties of the test such as how long to run the test for. You can easily configure different run settings for running the same test against different servers.

Load Test AnalyzerYou use the load test analyzer both during and after a run to analyzer the load test results. While the test is running, the analyzer displays performance counters collected from the load agents and system under test. You can control which counters are displayed on the graphs, as well as create new graphs. Once the test completes, you can also view a summary report as well as tabular data for the performance measurements collected during the test.

Once the test is complete, you can drill into specific areas of the test:

You can also view tabular data

And drill into details of requests that fail

Reporting
The summary report supports copy/paste to html format for easy sharing via email, web or msword.

The load test data can be published to Team Foundation Server which houses a summary report. All of the data is stored in SQL server, so you can use your favorite tool to report on the data. A powerful set of reports are available on codeplex.com/loadtestreports.

Scale Out Load Generation

With VSTS and VSTT you are able generate a significant amount of load. If you need to generate more load, you can run the load agent product on multiple computers to achieve scale-out load generation to many 1000s of users.

Resources and Community

There are many resources you can leverage to make your effort with these tools more effective. Ed Glas maintains a list of links which includes links to guidance, how to videos, blog posts, and various codeplex projects with complementary tools for load testing, including tools for WCF recording, SQL recording, and load test reports. There is also a very active community on our Web and load test forum.

Summary

As you can see, Web and load tests provide a rich and capable solution for your load testing needs. Give it a try today! You can download the VSTS trial here, which contains the web and load test features.

Comments

• Anonymous
  January 29, 2009
  PingBack from http://www.anith.com/?p=3677
• Anonymous
  February 04, 2009
  The annual testers choice awards are out in STP magazine , and VSTS took top spot in two categories in
• Anonymous
  February 05, 2009
  [ Nacsa Sándor , 2009. február 6.] Ez a Team System változat a webalkalmazások és –szolgáltatások teszteléséhez
• Anonymous
  February 08, 2009
  [ Nacsa Sándor , 2009. február 8.] Ennek a processzoronként licencelt szoftvernek a segítségével skálázni
• Anonymous
  February 08, 2009
  Hi Amit,I will look forward to try the load test features of VSTS.Thanks,Inder P Singh
• Anonymous
  February 13, 2009
  Amit Chatterjee, General Manager of the VS Test Business, recently started blogging and has a post that
• Anonymous
  February 16, 2009
  I am trying to designate a loadtest that incorporates multiple agents, where one agent runs a specific set of webtests and the other agents run other webtests.  Why can't the test mix set scripts by agent machine?  This would be easier than trying to create a plug-in to manage this.  Any workarounds that can be done to quickly set this up?
• Anonymous
  February 17, 2009
  We are just indroduced to Visual Studio Team Test Load System in our company.I want to know what all the counter I can set here. Like if I want see whether Network in slow or Application or may be problem is in my machine.How can I setup that.Also want to know I can find application's break point.Again how?Plaese reply.
• Anonymous
  February 18, 2009
  Folks,I am not sure about posting this as comment here. My apologies if this is not the right place.I am looking for a Performance tester with VSTS for a contract opportunity for my client in Milwaukee. Contract is till end of '09.Any help is appreciated.Tx.Arokiaamariadoss@cube3tech.com
• Anonymous
  February 23, 2009
  Mark, in our next release (dev10) we have added the ability to specify a list of agents per scenario. I'm curious, why do you want to control this?Ed.
• Anonymous
  February 23, 2009
  Amita, probably the best way to find the breaking point is by using a stepping load. Step the load up until you see throughput go down, response times go up. Or until the server starts generating lots of errors.By default the standard counters -- cpu, memory, disk, and network -- are included in each of the counter sets. There are thresholds set for cpu and memory. Network is somewhat trickier, you will need to look at the network counters to understand if you are reaching the capacity of the network card.Ed.
• Anonymous
  February 23, 2009
  Amita, one more thing, there's an article on the analyzer you'll find useful here: http://msdn2.microsoft.com/en-us/library/aa730850(vs.80).aspxEd.
• Anonymous
  February 28, 2009
  Hi Amit,I recorded my login page and when I played back it showed Object Moved Here. And not able to find out the user name and password parameters. Please help.
• Anonymous
  March 05, 2009
  Object moved :- Is this an error or a notification of the response redirection?How to export load test in .CSV file?. i would be greatful for the answer. It would be great if someone can tell the excat steps.
• Anonymous
  March 09, 2009
  In your article you mentioned that reports are saved in SQL server " All of the data is stored in SQL server, so you can use your favorite tool to report on the data ". I have a LoadTest project and how do I look which SQL server it is connecting to and saving the reports. The project was not created by me and I do not know how to create either. In fact the testers in this project, run the load test and from the report they pick only couple of columns for reporting. And my task is to automate the process of picking only couple of columns. I was suggesting of a .NET utility(a console app ) which will pick report details from SQL server and save to excel. Is there any easier way to do it. Please help.
• Anonymous
  March 22, 2009
  Hi,The way its mentioned i have recored the script, this is the first time that i am having using VSTS. When i replay the script without any changes i get an error as "OBJECT MOVED TO HERE" can anyone help me. wht is this error is all about???Thanks & Regards
• Anonymous
  March 22, 2009
  Hi Hema,Have you got the answers to which you have posted?? because iam also getting the same and iam using this tool for the first time. Can you Help me???Regards
• Anonymous
  March 28, 2009
  Hema, what you are reporting is a redirection and not necessarily an error.  Many times webforms causes this redirection when it does authentication.Could you post your question to our forum with any additional details about your error so you can have a broader range of people helping you with any issuesYou can find our forum here http://social.msdn.microsoft.com/Forums/en-US/vstswebtest/threads/#page:1
• Anonymous
  March 28, 2009
  Prasad, I am not sure what you are asking here. Are you looking to export the data to CSV?  If so you have some better options. Please see the reporting project on codeplex  http://www.codeplex.com/loadtestreportsAlso, see my comment on Hema's question posted just ahead of this
• Anonymous
  March 28, 2009
  Sarvotham, the SQL server is specified on the controller that you are running the load test in.  Please see http://msdn.microsoft.com/en-us/library/ms243400(VS.80).aspx on MSDN for instructions.For reporting I recommend you look at the reports we have on codeplex as a start http://www.codeplex.com/loadtestreports
• Anonymous
  March 28, 2009
  Rangarao, is not necessarily and error but it is an indication you have been redirected.  Most browsers handle this automatically so you never see it happen.  If you search our forum http://social.msdn.microsoft.com/Forums/en-US/vstswebtest/threads/#page:1 you can find a variety of reasons this might happen.  
• Anonymous
  March 31, 2009
  Hi, when i did the load test below mentioned is the errors which i got they R of nearly 203 error Count.Request failed: Context parameter '$HIDDEN1.__RequestVerificationToken' not found in test contextRequest failed: Context parameter 'FormPostParam3.MessagePreferencesModel.Preference.FundValuationChange' not found in test contextKindly let me know how to overcome such errors.Best WishesRitu
• Anonymous
  May 04, 2009
  hello, i have a problem using web tests in load test.All the results of the web test are 0. i have global results but when i want to check the results of each test i see all 0.please any ideas????
• Anonymous
  May 19, 2009
  I am very excited proud and proud to introduce the new testing features you’ll find in dev10. In addition
• Anonymous
  June 01, 2009
  Hi,Would anyone be able to shed some light on threshold violations for % Time in RT checks threshold, and how to configure them?Does this counter have substantial effect on the load testing?Thanks for the helpS
• Anonymous
  June 02, 2009
  Hi Amit,I’m using de Visual Studio for Tester and had a problem.I made a Web Service (WCF) that saves a new entry in data base.After, I made a UnitTest that calls the WebService and returns success if the return of WebService is different of null.After, I made a Load Test that have five simultaneous users (virtual) and only one interaction.I ran the load test with success, but I expected it would be recorded five entries in the database, one per user (virtual).But when I ran the load test, only one entry was recorded in data base.This behavior is correct? If yes, why?I searched for more information of architecture of the Load Test of Visual Studio, but don’t find.You have articles or links that explain how works the Load Test?Thanks.
• Anonymous
  June 09, 2009
  Hi ,Can anyone tell me what  is threshold violation?ThanksCathy
• Anonymous
  June 25, 2009
  hi amit,        i have one question that does vsts support java scripts controls ??
• Anonymous
  July 01, 2009
  Hi Amit,I am a beginer of VSTS Perf testing.Can you please tell me how to add counters (which Counter are more necessary) and how to analyse output of counters.Also any link to set up rig.
• Anonymous
  July 07, 2009
  hi amit,the blob is really helpfulbut i do need some more info,,can we change the browser in which web test is performed,some of the pages are not working in IE but working in firefox,,can u please help with this,,help is appriciated..
• Anonymous
  July 07, 2009
  hi amit,the blob is really helpfulbut i do need some more info,,can we change the browser in which web test is performed,some of the pages are not working in IE but working in firefox,,can u please help with this,,help is appriciated..
• Anonymous
  August 12, 2009
  The comment has been removed
• Anonymous
  October 11, 2009
  Hello Amit,I am new to software testing, so please apologize for my lack of knowlegde in the same.Scenario:Step 1:  A new user registers himself by supplying his information on a website and presses a submit button.Step 2: From the server side, an email is generated with a link to click for confirmation and sent to the user.Step 3: The user clicks the link and logs himself on to the site to confirm.Problem: I have to automate the scenario as webtest. Till the 1st step, everything was done well(I hope) by preparing a *.csv file with random usernames and binding it to the webtest so that the username's formpostparameters access the *.csv file and chose the usernames for registration randomly. Once a username is chosen and registered, an email is generated with a confirmation link so that the user clicks it and logs onto the website to confirm his registration. The link has a 16 characters long randomly generated ID(specific to the username). My problem is: how to integrate this link as a request into the webtest so that the webtest automatically supplies ANY username for registration, clicks the Email link, login the user for confirmation when ever the test runs.I have searched for this problem on internet but I could not found any. Please help me with Ideas and suggestions. By the way I am using Visual Studio 2008 professional.Thanks in advance for any kind of help.Kind Regards
• Anonymous
  October 15, 2009
  How can I simulate multiple windows user authentication whikle doing my load test ???
• Anonymous
  November 11, 2009
  I have an application to test the load for 200 users.i need test major functionalities as login and registration of their tasks.To do the load test for this application i have created the wen test. my question is to do the load for 200 users do i need to create 200 users and provide those ser names to the test ??
• Anonymous
  November 11, 2009
  I have an application to test the load for 200 users.i need test major functionalities as login and registration of their tasks.To do the load test for this application i have created the web test. my question is to do the load for 200 users do i need to create 200 users and provide those user names to the test ??
• Anonymous
  November 17, 2009
  Hi,I am using vsts.I want to ask....i have recorded an application using vsts.I replayed the script also.I want to see the script which was recorded to analyze the same. Where can i find it.ThanksDeepa.
• Anonymous
  December 15, 2009
  Hi,I have to perform a web test for a application which opens up new page for each function. how can I integrate all these new pages to the recorder of the web test.
• Anonymous
  February 01, 2010
  I have set up about 7-8 web tests to go against one of our servers, but I end up running it against 4 servers in total. It get VERY time consuming having to go through each test and change the parameterized webserver. Is there any way to have some time of global variable that I can change once and all the tests will run against the new URL?
• Anonymous
  March 18, 2010
  Hi Amit,Does test suite support Mozilla and Chrome. I have seen the list of other browsers that it supports
• Anonymous
  March 30, 2010
  Hi,I found this tool is useful for managing load test runs and creating test results.... http://www.qualiwebsolutions.com. Anyone used it?
• Anonymous
  April 19, 2010
  Hello Amit,I used VSTS for load testing, But page response time graph is not showing any data. Even the page response time is showing 0 in test result.'How can i check the page response time, will you please help me !
• Anonymous
  April 27, 2010
  Sravanthi - yes it does support Chrome. For unsupported browsers, you can go to ~Program FilesMicrosoft Visual Studio 10.0Common7IDETemplatesLoadTestBrowsers, and add a new template (use the existing ones as examples)
• Anonymous
  April 28, 2010
  what type of protocol does vsts support for performance testing
• Anonymous
  May 25, 2010
  really i need this information...and provide me a great idea to do the load testing.......I m searching this type of inforamation so that i would be able to perform load testing..to test my website that how many user would access it at a time in a server...thanks amit for sharing such information.
• Anonymous
  June 08, 2010
  Hi i am testing an application using vSTS. while recording, i dont get any error but while playback we get java script disabled error . Please help me to solve this issue
• Anonymous
  August 04, 2010
  Hi,I am using Visual studio 2008 Team System SP1. I need to perform Load Testing, but when I am going to add a "Load Test" after clicking the Add Tab only "Unit Test" and "Ordered Test" is comming not other options as I saw in the How Do I? videos.Can u plz help me out from here..Tools-->Options-->Test Tools-->Test Projectshere also in the right pen I am getting only two options "Unit & and Ordered" test.thanks in Advance
• Anonymous
  November 15, 2010
  Look and feel is change in 2010 VS and TFS
• Anonymous
  November 19, 2010
  Can we do it in the Visual Studio 2010 ultimate.
• Anonymous
  November 24, 2010
  Can we do a web performance test on web applications developed using Java, Informatica, ETL Modules, Microstrategy?
• Anonymous
  July 25, 2011
  thanks for sharingsoft-engineering.blogspot.com
• Anonymous
  January 22, 2012
  Hi Amit, Good Morning , First of all thank you so much for sharing your knowledge. I have one question. By using MSVS 2010 can we define a script as action block and save it to external library and then call it to different tests? please let me know if you have any information on this. Thanks.
• Anonymous
  January 24, 2013
  hi i want information on "How to test the performance of xap file(silver light) in visual studio ultimate.reply me to suresh1.dvm@gmail.com
• Anonymous
  March 11, 2013
  I'm not able to load a web test once it is saved . I'm getting error like this load test can not be loaded, The problem is only with our microstrategy application where as I'm able to save and open load test created for normal java applications in the same workspace and project. Please help in troubleshooting this problem
• Anonymous
  March 23, 2014
  hi,can anyone tell me how to insert validation  on vsts web testthanks
• Anonymous
  April 23, 2014
  Can i get videos related to performance testing