Advances in Windows Vista TCP/IP
The Windows Vista TCP/IP stack has made tremendous improvements in its efficiency, taking full advantage of hardware advances (e.g. gigabit networking). As explained by Murari in a previous posting (Advances in Windows TCP/IP Networking), there are a number of bottlenecks that affect TCP throughput. Here, I will give some examples of how we’ve addressed these bottlenecks in the Windows Vista TCP/IP stack.
TCP auto-tuning: At any given time, the amount that TCP can send is governed by three factors: the congestion window, the receive window and the number of bytes available to send. Without using TCP window scaling (which is disabled by default in previous versions of Windows), the maximum receive window a receiver can advertise is 64K bytes. Since the congestion window is usually greater than 64K bytes in high-bandwidth/high-latency networks, the receive window is often the limiting factor if the application is submitting enough data.
In previous versions of Windows, users can work around this problem by setting the TcpWindowSize registry key value. However, TcpWindowSize is a global setting applied to all connections, and it’s often hard for users to know the appropriate window size to set.
To address this issue in Windows Vista, we implemented TCP auto-tuning. It enables TCP window scaling by default and automatically tunes the TCP receive window size based on the bandwidth delay product (BDP) and the rate at which the application reads data from the connection. With TCP auto-tuning, we have seen 1000% (10x) throughput improvements in internal testing over underutilized wide-area network links.
Receive-Side Scaling: Networking stacks face a number of challenges in scaling their receive processing across processors on multi-processor systems. For instance, on previous versions of Windows all packets indicated in a single interrupt service routine (ISR) are typically processed in a single deferred procedure call (DPC) queued to a specific processor to avoid packet reordering. Until the outstanding DPC completes, no more receive indication interrupts can be triggered. As a result, only one processor can be used at any given time for processing received packets for a single network adapter.
Receive-side scaling (RSS) is our solution for this issue in the new networking stack: it enables parallelized processing of received packets on multiple processors, while avoiding packet reordering. It achieves parallelism by allowing ISRs to queue DPCs on multiple processors, enabling packet processing on multiple processors at the same time. It avoids packet reordering by separating packets into flows, and using a single processor for processing all the packets for a given flow. Packets are separated into flows by computing a hash value based on specific fields in each packet, and the resulting hash values are used to select a processor for processing the flow. Using TCP as an example, this approach ensures that all packets belonging to a given TCP connection will be queued to the same processor, in the same order that they were received by the network adapter.
TCP offload: Previous Windows releases already support network task offload for stateless per-packet operations (e.g. LSO, checksum offload etc). In Windows Vista, in addition to the offloads supported on previous Windows releases, we’ve also introduced support for TCP chimney offload. TCP chimney offload enables Windows to offload all TCP processing for a connection to a network adapter. Offloads are initiated on a per-connection basis, based on heuristics. Compared to task offload, TCP chimney offload further reduces networking-related CPU overhead, enabling better overall system performance by freeing up the CPU for other tasks.
We have also responded to customer feedback by making the Windows Vista TCP/IP stack much smarter and more adaptive in a number of scenarios. One such improvement we’ve made is to enable TCP black-hole detection by default in Windows Vista.
Historically, problems due to the presence of black-hole routers have been among the highest product support call generators for the previous Windows networking stacks. To understand why, it’s important to know that TCP/IP relies on ICMP packet-too-big error messages to discover the maximum transmission unit (MTU) for any given connection’s path, so that it can reduce the size of the packets that it sends if they’re too large. If a router along the path does not send back ICMP error messages, or if a firewall drops ICMP error messages, TCP will never find out that its packets are too big. As a result, it will retransmit the packets repeatedly with the same size, up to its maximum number of retransmissions and, when it gets no responses, it will terminate the connection.
Black hole router detection is a mechanism used in this scenario to automatically reduce the size of the packets sent for a connection, based on the current status of the connection, in the absence of feedback from ICMP packet too big error messages. This mechanism was disabled by default in previous versions of Windows, because previous approaches would often yield too many false positives, lowering the packet size unnecessarily and reducing performance. In Windows Vista, our improvements have reduced the likelihood of false positives and, consequently, minimized the adverse performance impact, enabling us to turn on black hole detection by default in the upcoming Beta 2 release.
There are many, many more innovations that we’ve made in the network stack, far more than I can write about in this one posting. Stay tuned for more…
Xinyan Zan
Software Development Engineer, TCP/IP Networking
Comments
Anonymous
June 09, 2006
Pity it dowsn't work. Adobe, Disney and about 1/2 the internet isn't accessable with Firefox. I wonder if M$ have tied BlackHole detection into the IE libraries??Anonymous
June 30, 2006
The comment has been removedAnonymous
January 26, 2007
I've been in converstations with consultants on some large global companies on determining which deploymentAnonymous
April 10, 2007
Наткнулся на замечательное руководство по оптимизации windows vista.Anonymous
November 19, 2007
The comment has been removedAnonymous
November 21, 2007
hey Al. If you'd like to submit a packet trace and tell us what router you are running, we'll take a look. Use the Email link at the top of the page to get communication going.Anonymous
November 28, 2007
The reason for your dropped packet is that Vista has the ability to increase the TCP window size to a 32-bit value. Since this is a relatively new option. Most firewalls, routers, etc. are not capable of dealing with these larger window sizes. I believe patches were made dealing with this issue.Anonymous
December 02, 2007
Could TCP Autotuning be the cause of a problem I keep getting with Windows Mail? Quite often I get an error code 0x800CCC19 and my server times out without sending mail. The only way I can fix this is to shut Windows Vista & then restart. Very annoying. This seems to happen irrespective of Firewall settings. I have read somewhere that disabling TCP Autotuning will fix this?Anonymous
December 13, 2007
C:>netsh interface tcp set global autotuninglevel=disabled Ok. C:>netsh interface tcp show global Querying active state... TCP Global Parameters
Receive-Side Scaling State : enabled Chimney Offload State : enabled Receive Window Auto-Tuning Level : disabled Add-On Congestion Control Provider : none ECN Capability : disabled RFC 1323 Timestamps : disabled
Anonymous
December 19, 2007
Hi, I was wondering if anyone could email me and maybe tell me how to get ecn enabled on my computer I have windows vista Home premium. I have a linksys router befsr41 v4.3 I also have a cable modem connection through Comcast I believe it's 768 package I only use the router as a hardware firewall that's it. I'm not running multiple computers or servers or anything like that.C:>netsh interface tcp show global I ran this and recieve side scailing is enabled chimmney offload state is enabled. The auto tuning for tcp is disabled they recommended to do that. Add on congestion controller provider is ctcp. ecn capability is disabled. I can't figure out how to enable it in my router that is through windows I do and windows take it but, I think both windows vista and my router have to support it. Also rfc 1323 timestamps is enabled. so any help you could give me would be awsomme thank you so much SteveAnonymous
December 29, 2007
So how do we set MaxMTU? An application that I'm running requires a MaxMTU of at least 1364 in order to function. How do I change that setting in Windows Vista?Anonymous
December 31, 2007
hi, I'm running Vista home premium 1GB RAM connecting to the internet via wireless linksys router connected to Comcast Cable internet. All of the other computers connected to the same router are running either XP home, or XP pro, and have no internet/network issues. constantly through the day, i have to turn off, and back on my wireless adaptor on this laptop in order to reconnect. it is wreaking havoc on everything I do with the internet from chatting to playing online games to surfing the web. I've changed the autotune feature moments ago, and I am going to let you all know if it works for me... Thank you.Anonymous
December 31, 2007
The comment has been removedAnonymous
January 02, 2008
Jason: If the other machines are not having issues, I would check a couple things: a) Are you using the same wireless security settings on Vista as you are on XP? b) Check for updates for both the firwarm for your router and for the drivers on the wireless card in your computer. Christopher: As far as I understand the default MaxMTU on a ethernet network is 1500. You can see the current setting via "netsh int ipv4 show interfaces"Anonymous
January 02, 2008
The following enables ECN, but why do you want to set it? netsh interface tcp set global ecncapability=enabled Disabled, or default: netsh interface tcp set global ecncapability=default netsh interface tcp set global ecncapability=disabled and since this is the second question around this area, check out netsh interface tcp set global ? for help on how to set the various machine wide TCP settings.Anonymous
January 07, 2008
I just disabled rss and autotuning just so I could map a drive to Windows 2003 servers without waiting 20-30 seconds for each click to take complete. The window would show the contents but an indicator bar in the folder name space in explorer would take 20-30 seconds to complete whatever it was doing before I could actually access the files. Now everything is nice and snappy like Windows XP and Windows 2003. What exactly is the autotuning feature attempting to do every time I open a folder? Shouldn't it autotune the initial session once and leave it alone? The constant autotuning seems redundant and does not work in practice at least with my switches and seriously interfered with streaming audio and video even on my gigabit network. If this is a limitation of my switch or a compatibility issue then it should be noted but I have never heard of it until today.Anonymous
January 19, 2008
Plz, plz plz plz plz plz will you tell me how to enable my rss....Every where i look I see..... Receive-Side Scaling State : enabled Chimney Offload State : enabled Receive Window Auto-Tuning Level : disabled Add-On Congestion Control Provider : none ECN Capability : disabled RFC 1323 Timestamps : disabled BUT MINE IS..... Receive-Side Scaling State : DISABLED Chimney Offload State : enabled Receive Window Auto-Tuning Level : disabled Add-On Congestion Control Provider : none ECN Capability : disabled RFC 1323 Timestamps : disabled .... Also whnever I open an internet page it pauses ofr about second first... Please help Thank you You want:netsh int tcp set global rss=enabledRegarding IE startup time, check out this troubleshooting page. -- AriAnonymous
January 27, 2008
We are running 2 Vista machines & 1 still on XP (the XP for 5 years with no problems.) Also running 1 linux machine with similarly trouble free E-mail via the same servers. On the 2 Vista machines , I am continually getting Mail Server time-outs with error code 0X800CCC19. Have tried various fixes all to no avail. Have to restart the PC everytime it happens. Why does Microsoft continue to introduce expensive new products which simply refuse to work in the real world?Anonymous
February 21, 2008
How do i change the the TCP recieve window when autotuning is off? TCP optimizer and dr.tcp dont work. Apparently, you don't. The registry key applied to all interfaces (not a good thing), we believe that autotunning is a much better model. Is there a specific scenario you are trying to achieve? -- AriAnonymous
March 25, 2008
The comment has been removedAnonymous
April 01, 2008
The comment has been removedAnonymous
April 21, 2008
Hi - I am working on a server farm at the moment - running Win2003 Sp2 - We would like to increase the window size to optimise the server - server communications. We have 2 Web 1 Application and a 2 node SQL Cluster. All Servers communicate over a 1Gb Full Duplex Private network. They are used to host a MOSS2007 implementation - can you recommend the best window sizes for these boxes? - I have been struggling to find anything meaningful to make sure that our implementation real flies. Kind Regards, Nick While I don't have specific recomendations, you can look at KB224829 and sections of The Cable Guy for what you can tweak to find optimal settings. -- AriAnonymous
May 08, 2008
Server Farm Nick - I've done some testing with WS03, offer an "it depends" recommendation. Objective is to get improvement without exhausting non-swappable memory:
- 256 kByte windowsize & globalmax windowsize.
- Timestamps on for web servers, not necessary for app & DB servers (if you tolerate specialized tuning)
- Selective acks on
- Reduce TcpTimedWaitDelay to 30 seconds to free memory more quickly
- Modern NICs offer a lot of offload options - see advanced properties. Use them. Also increase the number of Receive & Transmit Buffers. Notes:
- Reboot required for TCP tuning
- Apply equivalent regedits to client devices (except Vista, WS08 and Mac Leopard) for full benefit. Most people report their home broadband gets faster, too.
- WS03 x64 with 4 GB memory or more will support 1 MB max windowsize. This is useful for really broad band (1 Gbps WAN, 20+ mbps across oceans), provided you can apply equivalent tuning on the other end.
- Some afd (winsock) parameters are so small one would think they haven't been adjusted since W2k called 64 MB memory a large server. Adjust at your own risk, and lab test before production deployment. Rant:
- Ping time is only part of latency. Consider the time required for the receiver to process the info (clear receive buffer) and sender to tee up the next buffer-load of bits. iPerf doesn't include this.
Anonymous
July 20, 2008
Why when using Windows Mail my router losses connect to the internet? This only happens on my 2 Vista PC. Never had this issue on XP. After I do a check mail and I receive my mail, my router losses internet connectivity. I had to unplug the router to fix this. Any answers out there? emailto:aimperato@acn.netAnonymous
August 14, 2008
IF you need a easy smart vpn sultion, tell your IT to check out OpenVPN. if you have ? about it, goto http://openvpn.net/ . its free and open sourceAnonymous
August 14, 2008
I agree Vista needs alot of work, but not really. See Vista is here for windows server 2008. Not server 2003. Its to get everyone off of XP and into vista. Just like when Support ended for win98. It was making a shift to XPAnonymous
August 29, 2008
I am trying to find an easy solution to setting up an ethernet (crossover cable ) between a vista SP1 PC and a Mac with OSX Leopard. Mac sees PC files (wont print share yet!) and I can transfer, but vista receives transfers but the Mac will not show up in vista explorer no matter what I do! Any help appreciated. If transfers are actually happening, it sounds like the issue isn't in networking, but prehaps in something else. I don't personally use OSX, so I'm not sure what could be happening.... If I shake my magic eight ball, I get "investigate the file types and expected location of the share". -- AriAnonymous
September 03, 2008
hi, Could you please let me know how you have the receive window size increased in the Vista machines. I know it has the feature to auto tune the RWIN and window scaling but no matter what options I use in netsh I get only about 1.7Mbps. Where as in windows 2000 and XP i can get 6Mbps and more because I am able to set the parameters in the registry. Now I see that Vista PC is always keeping the window size to 64K (with scaling 8K x 8). How can this behavior be changed. I need to have larger window size for my setup. ThanksAnonymous
November 11, 2008
We like many other people are trying to see how we can hardwire the TCP recieve window to something which we know if optimised for the network we are running. If we cannot be allowed to do this then we will just have to take this global application into Linux as we would have been too tightly controlled by microsoft in a way that stops us from working.Anonymous
November 19, 2008
Andrew: The application always has the choice to specify its own receive buffer via the SO_RCVBUF socket option. When this socket option is specified, autotuning will no longer affect the socket. Now if autotuning doesnt provide optimal performance for your network, I would be happy to take a look at it and see how it can be improved. Could you please obtain network captures (using ethereal or netmon) of the case showing poor throughput, perferably taken at both the sender and receiver? Thanks VivekAnonymous
December 10, 2008
go http://ubuntu.com and be happy And now you have two problems... :) -- AriAnonymous
February 02, 2009
Sorry for a bit off-topic, but maybe somebody here knows to what advance in Vista we owe that ICMP raw sockets do not recieve TTL expired or UNREACHABLE type ICMP replies, and how to workaround that one?Anonymous
February 17, 2009
The comment has been removedAnonymous
April 28, 2009
John, re your vpn connection,it can be the IPSEC keepalives. check your client to see if you can disable the keepalives all-together. I know that the nortel and checkpoint clients support disabling the keepalives.Anonymous
June 01, 2009
Any pointers to guidance about the virtues of enabling / disabling TCP timestamps?Anonymous
February 22, 2010
Hi Xinyan, I was reading your blog with interest. would you be able to help. My vista reliability and performance monitor shows 0 percent network utilization and the bandwidth is 1MB. When I download a file the speed is 10 percent(101kbps) of the 1MB. I never get anywhere near the 1MB bandwidth. I believe it is a problem in vista as my ISP confirms the speed up to the point of my router is 1MB. I have looked at and tried solutions such as running the netsh command, disabling remote compression differential in vista, running vista in safe mode with networking but have had no improvement. Please can you suggest something. Thanks. chris.Anonymous
January 04, 2012
Hello, what are the recommendations for a virtual machine running in hyper-v? Since the NIC is virtual and not physical, shouldn't we disable all of these features?