HOWTO: Diagnose IIS6 failing to accept connections due to Connections_Refused
Recently, I have seen a bunch of questions asking: "IIS6 runs fine for X amount of time [where X varies from days to weeks] and then all of a sudden, it stops accepting all connections. If I restart/reboot the server, then it is fine again... until X amount of time later, when I have to repeat the same procedure."
Sometimes, the user even notices that HTTP Error logs in %SYSTEMROOT%\System32\LogFiles\HTTPERR\*.log mention Connections_Refused for all those requests, but what the heck does that mean?
Ok, ignoring the jokers in the back that are snickering "hmm, this is Microsoft software; what do you expect?" , this issue is actually very diagnosable and should be treatable, as I will shortly show...
Question:
Every four or five days the web site hoste on my machine will stop responding to HTTP requests on port 80.
IIS doesn't crash so I cannot get an IIS State log, just this:
Excerpt from the HTTPErr.Log:
(... working fine up until this ...)
2005-09-02 18:45:12 - - - - - - - - - 1_Connections_Refused -
2005-09-02 18:46:07 - - - - - - - - - 1_Connections_Refused -
2005-09-02 18:46:27 - - - - - - - - - 2_Connections_Refused -
2005-09-02 18:46:42 - - - - - - - - - 1_Connections_Refused -
2005-09-02 18:47:02 - - - - - - - - - 2_Connections_Refused -
2005-09-02 18:48:22 - - - - - - - - - 2_Connections_Refused -
2005-09-02 18:48:57 - - - - - - - - - 2_Connections_Refused -
2005-09-02 18:49:02 - - - - - - - - - 1_Connections_Refused -
2005-09-02 18:49:42 - - - - - - - - - 7_Connections_Refused -
2005-09-02 18:50:02 - - - - - - - - - 3_Connections_Refused -
2005-09-02 18:50:37 - - - - - - - - - 6_Connections_Refused -
2005-09-02 18:50:42 - - - - - - - - - 2_Connections_Refused -
2005-09-02 18:51:02 - - - - - - - - - 2_Connections_Refused -
2005-09-02 18:51:47 - - - - - - - - - 1_Connections_Refused -
2005-09-02 18:52:02 - - - - - - - - - 3_Connections_Refused -
2005-09-02 18:52:27 - - - - - - - - - 1_Connections_Refused -
2005-09-02 18:53:17 - - - - - - - - - 1_Connections_Refused -
2005-09-02 18:53:22 - - - - - - - - - 2_Connections_Refused -
2005-09-02 18:54:02 - - - - - - - - - 2_Connections_Refused -
2005-09-02 18:54:22 - - - - - - - - - 1_Connections_Refused -
2005-09-02 18:54:32 - - - - - - - - - 1_Connections_Refused -
2005-09-02 18:55:22 - - - - - - - - - 1_Connections_Refused -
2005-09-02 18:56:02 - - - - - - - - - 2_Connections_Refused -
2005-09-02 18:56:42 - - - - - - - - - 1_Connections_Refused -
2005-09-02 18:58:17 - - - - - - - - - 2_Connections_Refused -
2005-09-02 18:58:42 - - - - - - - - - 2_Connections_Refused -
2005-09-02 18:58:57 - - - - - - - - - 1_Connections_Refused -
I have to reboot the machine to get it to work again.
Does anyone have any ideas?
Answer:
Connections_Refused is actually a very diagnosable failure pattern, and I am going to show you how to diagnose it and what is going on.
What does Connections_Refused Mean?
According to KB 820729, Connections_Refused means that the "Kernel Non Paged Pool memory has dropped below 20MB and HTTP.SYS has stopped receiving new connections". What all this means in English is this:
On Windows NT systems, there are many types of memory with different properties. Kernel Non Paged Pool (NPP) memory is one such type, and the important thing to know is that on x86 machines, the size of this memory is fixed and has a maximum size of 128MB. In other words, you may have an x86 machine with 4GB of RAM, but if you only use 128MB and that 128MB all comes from NPP memory, your system is still "out of memory" regardless of how much is actually in use.
[1/4/2006 - Correction - maximum size on x86 is "around 250MB" and 128MB with /3GB. For Windows Vista, it will be bounded by physical RAM and not arbitrary limit.]HTTP.SYS, being a kernel-mode driver, uses NPP memory for every single connection that is active, and we made a conscious decision to have it stop accepting connections when NPP memory falls below a threshold, arbitrarily set at 20MB. This is reasonable because lots of bad things can occur if NPP memory is depleted.
For example: HTTP.SYS is likely the most active codebase in a web server, and if NPP memory gets depleted and HTTP.SYS needs NPP memory to accept a connection, the machine will blue screen. Now, regardless of the driver leaking NPP memory, HTTP.SYS looks like the culprit because it is most active. In any case, all of this is not good, so HTTP.SYS proactively refuses to get near the edge and plays it safe.
Hopefully, it is now clear what Connections_Refused means. HTTP.SYS is basically telling you "someone is using up a lot of NPP memory, and for protective reasons, I am going to stop accepting requests". We need to figure out what driver is using up all the NPP memory and address it, and the Connections_Refused should naturally go away because NPP memory will not be under pressure.
How to Diagnose this Issue
One of the nice properties of NPP memory is that a unique tag is associated with every piece of this memory, so we can always track down who is using what and how much of NPP memory. The user-mode tool to do this is called POOLMON.EXE (if you have a Kernel Debugger attached, you can use the !poolused command to get this info within the Kernel Debugger). Here are some additional bits of related information:
- KB 177415 - How to Use Memory Pool Monitor (Poolmon.exe) to Troubleshoot Kernel Mode Memory Leaks - since Windows Server 2003 has PoolTagging enabled by default, things just work.
- KB 298102 - How to Find Pool Tags that are used by Third-Party Drivers - this is basically searching for a NPP tag string within all drivers on the system hoping for a logical match...
I am now going to step through the POOLMON output of a user who had Connections_Refused and comment on what is going on. Here is a sample snippet of running POOLMON.EXE -b :
Memory: 1048016K Avail: 438396K PageFlts: 219 InRam Krnl: 3604K P:42372K
Commit: 538908K Limit:11245092K Peak: 609576K Pool N:109056K P:43740K
System pool information
Tag Type Allocs Frees Diff Bytes Per Alloc
ISil Nonp 1408366 ( 56) 1327427 ( 58) 80939 36678632 ( -872) 453<br> I100 Nonp 11048877 ( 217) 10967968 ( 219) 80909 14886928 ( -368) 183
PcNw Nonp 292427 ( 0) 88967 ( 0) 203460 12203296 ( 0) 59
SavE Nonp 1596066 ( 0) 1595659 ( 0) 407 11717016 ( 0) 28788
IAM Nonp 1946164 ( 49) 1904725 ( 50) 41439 8519576 ( -208) 205
tdLL Nonp 74748 ( 0) 69830 ( 4) 4918 2672832 ( -1776) 543
LSwi Nonp 1 ( 0) 0 ( 0) 1 2576384 ( 0) 2576384
R200 Nonp 22 ( 0) 1 ( 0) 21 2297816 ( 0) 109419
MmCm Nonp 302 ( 0) 48 ( 0) 254 2236960 ( 0) 8806
TCPt Nonp 190486 ( 4) 190462 ( 4) 24 1392264 ( 0) 58011
ULHP Nonp 3127 ( 0) 41 ( 0) 3086 1047216 ( 0) 339
PTrk Nonp 999750 ( 0) 996561 ( 0) 3189 956824 ( 0) 300
File Nonp 9628160 ( 65) 9623252 ( 65) 4908 750688 ( 0) 152
rg81 Nonp 6910 ( 0) 5294 ( 0) 1616 603864 ( 0) 373
Pool Nonp 6 ( 0) 3 ( 0) 3 602112 ( 0) 200704
Devi Nonp 680 ( 0) 262 ( 0) 418 558552 ( 0) 1336
Thre Nonp 417221 ( 6) 416457 ( 6) 764 476736 ( 0) 624
LSwr Nonp 128 ( 0) 0 ( 0) 128 416768 ( 0) 3256
Mm Nonp 26 ( 0) 2 ( 0) 24 379880 ( 0) 15828
AfdC Nonp 21989 ( 0) 19872 ( 1) 2117 338720 ( -160) 160
....
Some observations:
- Notice that this system has 1GB RAM but still only 128MB NPP memory (109MB has been used, hence tripping the 20MB marker of HTTP.SYS).
- The NPP memory tags of HTTP.SYS all start with "UL" (for the trivia-inclined: as with all software projects, HTTP.SYS has gone through some evolution of its own. It used to be called something else - Universal Listener - hence its tag names all started with "UL").
- Notice that HTTP.SYS is not even in the top-10 NPP memory usage on the system, and its largest set of allocations is barely using 1MB total. As it turns out on this system, TOTAL NPP memory usage by HTTP.SYS was under 2MB, so clearly, it is not the aggressive user of NPP memory that is now causing issues.
- Meanwhile, focus your attention on the top four users of NPP memory, with tags "ISil", "I100", "PcNw", and "SavE". I looked up drivers for those tags, and I only find that "PcNw" comes from Microsoft and is associated with WDM audio. The others are all 3rd party drivers.
- Umm, this is a server, so is audio really that important? I checked on my system, and it was at 275KB, so clearly this user is/has done some heavy-duty audio stuff on this server. Thus, the memory may be legitimately used. You know, you might want to reconsider the effects of playing your MP3s on your Web Server's ability to accept connections...
- Meanwhile, Isil and I100 have suspiciously similar active allocations and ISIL is the stock ticker for a semiconductor company, so my guess is that I100 is related to a Fast Ethernet card... and since web servers totally rely on the network driver, having a memory-hog for a network driver cannot be a good thing and should be fixed. My guess for SavE is probably anti-virus.
- Now, I may be totally wrong with those arbitrary tag names, but one thing is for certain - IIS6 and HTTP.SYS is running just fine on this system. There are some other memory-hungry drivers running on the system, and if you get fixes for them such that your system will stay running longer, IIS6 will likely keep running as well
Note: Don't shoot the messenger (HTTP.SYS)! Always strive to figure out the root of your problem and address that. The rest of the problems should naturally fall away.
//David
Comments
Anonymous
September 22, 2005
Great article. Also you can suggest the user to use Poolmon -c (2003 DDK) to create a "localtag.txt" (if not present) to find out the driver.
Cheers,
ArmandoAnonymous
September 22, 2005
Armando - Thanks. POOLMON -c is a good tip to map back to the appropriate driver name, which can hopefully provide some more contextual info...
//DavidAnonymous
September 24, 2005
Hi Dave,
I am having a very similiar problem with IIS 5 running .Net 1.1 webservices on Win2k. It is refusing connections in much the same manner as you describe. I am not getting any log information at all. The information I have been able to collect indicates to the client calling the web service that the connection has been refused. The behavior is sporadic, but seems to follow a pattern of worsening over time.
Does anything you have written thus far about IIS 6 apply? What can I do to troubleshoot this problem on IIS 5 since it doesn't seem to have an HTTPERR file? Any help at all would be vastly appreciated.
I'm not an IIS administrator, but a software architect attempting to rule out the software as the culprit.
Thanks,
David CampbellAnonymous
September 24, 2005
David - Unfortunately, since IIS5 and IIS6 are completely different architecturally, what I have said above only applies to IIS6 and has no applicability to IIS5.
We completely rewrote IIS6 and then worked on behavior-compatibility with IIS5, so you are mostly looking at different issues.
I would contact Microsoft PSS for paid incident support. Of all the times I have seen the similar sort of issues on IIS5, it was usually something wrong with software on IIS5, including Exchange.
And if it is a bug in MS software under support, you will not be charged and should expect to either get a fix (if it is known) or get an analysis of what is wrong and possibly add a fix-request into the service pack queue.
//DavidAnonymous
October 25, 2005
Thanks for this entry in your blog! I was able to find the application at fault -- Executive Software Undelete (tag is XEFr where X equals 1, or 2, or 3, etc). Once the service was stopped, which was number 1 on the list of tags shown, I was then able to hit pages in the website.Anonymous
October 25, 2005
Amy - glad to hear that you were able to get to the bottom of things.
You see, it is very easy to run "bad" software and make it look like something else is at fault. Only by figuring the issue out do you get a good sense of what is good or not.
There is a long history of IIS being the focal point of such unwarranted blame (not to say that IIS is completely blemish free, but it certainly picks up a lot of excessive slack). I am just doing my little part in helping people figure out what is really going on so that they can make their own decisions.
//DavidAnonymous
December 10, 2005
We have a similar issue on Windows 2K as David Campbell. 120 website hosting Asp and ASP.NET. After some period of time we get: The server was unable to allocate from the system nonpaged pool because the pool was empty.Anonymous
December 10, 2005
We have a similar issue on Windows 2K as David Campbell. 120 website hosting Asp and ASP.NET. After some period of time we get: The server was unable to allocate from the system nonpaged pool because the pool was empty.Anonymous
December 11, 2005
Ryan - You can use GFLAGS to enable "Pool Tagging" and then use POOLMON to see what is using up your NonPaged Pool memory and go from there. Since IIS5 and IIS6 are different architecturally, I can only start from "what is using up NonPagedPool Memory" and work backwards from there. Unlikely for IIS5 to be directly involved since there is no HTTP.SYS.
I believe GFLAGS is in the Microsoft Debugging Toolkit ( http://www.microsoft.com/whdc/devtools/debugging/default.mspx ), while POOLMON is in the Windows Resource Kit, both freely available.
//DavidAnonymous
December 12, 2005
Thanks David. I am doing that. Here is the top of my detail:
Memory: 3997212K Avail: 2716404K PageFlts: 1248 InRam Krnl: 2480K P:270528K
Commit:1097260K Limit:5900908K Peak:1294344K Pool N:102460K P:270732
Tag Type Allocs Frees Diff Bytes Per Alloc
File Nonp 87044825 ( 768) 86990579 ( 774) 54246 8786880 ( -992) 161
MmCm Nonp 1559 ( 0) 7 ( 0) 1552 7336512 ( 0) 4727
MmCa Nonp 991170 ( 1) 940356 ( 5) 50814 6440160 ( -512) 126
Ntfr Nonp 730966 ( 0) 632909 ( 0) 98057 6276640 ( 0) 64
tdLL Nonp 361780 ( 0) 351199 ( 27) 10581 6076736 (-11520) 574
NtFs Nonp 799244 ( 3) 747632 ( 6) 51612 3305600 ( -192) 64
Ntfn Nonp 440263 ( 0) 388659 ( 2) 51604 3304576 ( -128) 64
LSwi Nonp 1 ( 0) 0 ( 0) 1 2293760 ( 0) 2293760
PTrk Nonp 164833 ( 0) 161574 ( 0) 3259 1051520 ( 0) 322
It does not look like there is a major offender. Just we have a lot going on. I think I need to change the size of NonPagedPoolSize. This link here explains how to do it: http://www.microsoft.com/technet/prodtechnol/windowsserver2003/library/DepKit/fe862e6b-14fb-4ad1-bd76-7e409c37060d.mspx but does not tell me much detail. Any idea what happend to http://support.microsoft.com/?kbid=126402 i have foudn many references to it. But the article seems to be gone.Anonymous
December 12, 2005
Ryan
We are running IIS 5 on four Web server at my organization, and are having the same issue with degraded performance over time on two of them.
We also get an event logged that the NPR pool being empty before the server stops completely and must be rebooted.
I have been monitoring nonpageable ram in both the PROCESS and MEMORY counters, and have seen that the process counters are stable while the MEMORY (total NPR) increases steadily over time. This tells me that a regular process does not have a memory leak.
I have been watching poolmon for a few days and have noticed one tag in particular appears to be growing and not releasing memory, and this tag happens to be on your list as well: tdLL.
I believe this belongs to Symantec (the string is found in three of their sys files). Are you running Symantec A/V? We are running Corporate version 9.0.0.338.
Anyway, watch tdLL in poolmon over a few days and see if it gets bigger and bigger.
I'll post here if I find anything new.
David SummersAnonymous
December 13, 2005
David Summers,
Yes, tdLL was at the top of my list and it was the only one that I had been unable to identify. It has grown over time from 1,405,440 bytes initially (Saturday)to 10,555,008 bytes currently (Tuesday). Yes we are running Symantic AV Corporate version 9.0.1.1000.
Ryan M. HagerAnonymous
December 17, 2005
OK, one week later. tdLL is 70,586,816.
Yes, whatever it is, that is my culprit. FYI, we are using a Compaq server. And have diskeeper on top of NAV.
Ryan.Anonymous
December 20, 2005
Great article David. One minor thing though, according to Russinovich & Solomon's "Windows Internals", the maximum nonpaged pool size on a 32-bit system is 256mb, although 128mb if booted with the /3gb switch enabled in the boot.ini.
Here is an easy way to determine max npp size on a Server 2003 system:
1. Download and install the debugging tools from:
http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx
2. Download Process Explorer from:
http://www.sysinternals.com/Utilities/ProcessExplorer.html
3. Open Process Explorer and select "Options" then "Configure Symbols..."
4. Put the path to the "dbghelp.dll" included with the debugging tools you downloaded.
5. Use the following for the symbols path ("c:symbols" can be whatever...):
srvc:symbolshttp://msdl.microsoft.com/download/symbols
6. Select "View" then "System Information". You can now see the "Nonpaged Limit".
-MikeAnonymous
December 20, 2005
Mike - Thanks for the info.
I got that number from the HTTP.SYS folks who swear by those numbers (including the dev that actually implemented the check that results in Connection_Refused), so I'll report this bit of info to them.
//DavidAnonymous
January 04, 2006
Just a confirmation on what Mike was saying. I was just diagnosing the connection refused error today and came across your blog (very helpful by the way). I currently have a 206MB Nonpaged kernel limit with 189MB usage causing the problem.
Thanks for the information, it was very insightful.
-RichAnonymous
January 17, 2006
Well, After about a month we hit the limit again. Here is what we had this morning:
~8:00 am
Memory: 3997212K Avail: 1619692K PageFlts: 621 InRam Krnl: 2668K P:134468K
Commit:2298928K Limit:5569132K Peak:2566476K Pool N:252224K P:134656
Tag Type Allocs Frees Diff Bytes Per Alloc
tdLL Nonp 5767444 ( 7) 5483265 ( 13) 284179 163683072 ( -320) 575
File Nonp 1342225323 ( 194) 1342172019 ( 178) 53304 8559040 ( 2816) 160
AfdC Nonp 31320130 ( 8) 31276564 ( 7) 43566 8364672 ( 192) 192
And after an IISReset:
9:20 am
Memory: 3997212K Avail: 3042456K PageFlts: 1794 InRam Krnl: 2684K P:95548K
Commit: 742308K Limit:5569132K Peak:2566476K Pool N:215448K P:95740K
Tag Type Allocs Frees Diff Bytes Per Alloc
tdLL Nonp 5781423 ( 4) 5581628 ( 0) 199795 115072064 ( 512) 575
MmCm Nonp 1559 ( 0) 7 ( 0) 1552 7336512 ( 0) 4727
I guess it is time to give support a call.
- Ryan.Anonymous
January 19, 2006
Ryan
I uninstalled Symantec 9 A/V and installed Symantec 10 last week. My NPR leak has gone away, and the tdLL flag no longer contiuously grows.Anonymous
January 26, 2006
David S,
Yes, we upgraded today and have a new Memory Tag SavE (My guess Symantic Antivirus). I think this is even better so we can see more clearly if AV is the issue.
Memory: 3997212K Avail: 3208816K PageFlts: 986 InRam Krnl: 2504K P:64672K
Commit: 620408K Limit:5569124K Peak: 621072K Pool N:46768K P:64884K
Tag Type Allocs Frees Diff Bytes Per Alloc
SavE Nonp 747930 ( 120) 747490 ( 120) 440 13866816 ( 0) 31515
MmCm Nonp 1558 ( 0) 7 ( 0) 1551 7316032 ( 0) 4716
File Nonp 858782 ( 320) 842389 ( 331) 16393 2863104 ( -1664) 174
LSwi Nonp 1 ( 0) 0 ( 0) 1 2293760 ( 0) 2293760
tdLL Nonp 21511 ( 10) 18486 ( 22) 3025 1727168 ( -1984) 570
MmCa Nonp 54316 ( 26) 43039 ( 23) 11277 1437152 ( 384) 127
Ntfr Nonp 22473 ( 6) 158 ( 0) 22315 1429152 ( 384) 64
NDpp Nonp 54 ( 0) 24 ( 0) 30 981248 ( 0) 32708
PTrk Nonp 11485 ( 0) 8550 ( 0) 2935 947680 ( 0) 322
Ryan.Anonymous
March 02, 2006
The comment has been removedAnonymous
March 02, 2006
Ben - Thanks. Glad to help avoid some PSS incidents. Believe me, MS does not want you to call support, either. :-) They cost everyone money.
This particular issue and resolution is amazingly common, though the symptoms as reported by users can vary dramatically.
//DavidAnonymous
March 14, 2006
I recently sat down and thought a little about the typical user experience when troubleshooting IIS6,...Anonymous
March 14, 2006
The comment has been removedAnonymous
March 15, 2006
Mathias - try POOLMON -c
It should return a file named localtag.txt which associates the tag ID with a driver filename.
Given a filename, it is now up to your ingenuity to figure out what program/hardware actually installed and uses that file.
Good Luck,
//DavidAnonymous
March 15, 2006
It says, I need some dlls to create local tag file. Looks like i need the ddk for Server 200, or?
Costs me shipping. Is there a way to dl exactly these dlls, that i need?Anonymous
March 16, 2006
Well,
I 've been troubleshooting this problem for a while and I've seen this on two servers I've setup.
On both servers I was getting connection refused about every 6 days.
One Server is Small Business Server 2003 Standard and the other is Small Business Server 2003 Premium.
On the Standard I stopped the Symantec Veritas's Continuous Protection Server and they have been up for 29 days with no problems. I have SAVCE installed as well ver 10.0.2.200
On the premium server I was running fine until I loaded Symantec Veritas's Continuous Protection Server. So I figured no problem I'll stop the services associated with CPS like I did on the Standard. Well after 6 days they went down again. So not sure on this one yet. I have on theory but I will post a follow up after I verify it. The one tag that I'm not sure of on this server is QAFC (I think it's Quickbooks Enterprise 2006) I also have SAVCE installed on this server version 10.0.2.210
Memory: 4193360K Avail: 1582424K PageFlts: 517 InRam Krnl: 4368K P:68992K
Commit:2460984K Limit:10314256K Peak:2493512K Pool N:51056K P:70308K
System pool information
Tag Type Allocs Frees Diff Bytes Per Alloc
MmCm Nonp 576 (0) 14 (0) 562 12955920 (0) 23053
QaFc Nonp 5588 (0) 0 (0 45588 10211712 (0) 224
LSwi Nonp 1 (0) 0 (0) 1 2740224 (0) 2740224
File Nonp 22779779 (216) 22766589 (224) 13190 2011088 (-1216) 152
VoSm Nonp 811 (0) 783 (0) 28 1679600 (0) 59985
TCPt Nonp 90493 (6) 90463 (6) 30 1456496 (0) 48549
RxTi Nonp 18076337 (132) 18070773 (135) 5564 1290848 (-696) 232
Thre Nonp 818496 (22) 816639 ( 26) 1857 1158768 (-2496) 624
RadAnonymous
March 16, 2006
I haven't solved it yet but I'm optimistic. I found the QAFC is part of Qafilter.sys. I found this article first http://support.microsoft.com/default.aspx?scid=kb%3Ben-us%3B840141 but it only applys to Microsoft Windows Storage Server 2003.
Then I came across this http://seer.support.veritas.com/docs/276469.htm
I forgot that I had Veritas Storage Exec installed as well. So hopefully this will fix my problem since it fixes the memory leak in Qafilter.sys. I will let y'all know.
RadAnonymous
March 16, 2006
Mathias - I just performed a SIMPLE search for POOLMON on microsoft.com and came across several links providing download and other support information, such as the following:
http://technet2.microsoft.com/windowsserver/en/library//0d302498-c947-4655-95af-719ae75acfb51033.mspx
Interesting information include:
- POOLMON -c does not work on 64bit editions of Windows Server 2003
- the required DLLs for operation
//DavidAnonymous
March 17, 2006
Mathais - I ran into this same problem, unfortunately it does not provide a location to find them. I end up installing the following so that I could use poolmon /c instead of the -c (same thing from what I understand). After installing them all, I copied the
msdis130.dll and pooltag.txt files from the resource kit folder to the support tools folder. Then it worked.
Debugging Tools
http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx
Support Tools for 2K3 SP1
http://www.petri.co.il/download_windows_2003_sp1_support_tools.htm
Resource Kit for 2K3
http://www.microsoft.com/downloads/details.aspx?FamilyID=9d467a69-57ff-4ae7-96ee-b18c4790cffd&displaylang=en
I hope that this helps you,
RadAnonymous
March 17, 2006
Mathais - I ran into this problem as well. Unfortunately all the documentation on poolmon doesn't tell you where to find the files.
I had to install the following and then copy the msdis130.dll and pooltag.txt from the resource kit to the support tools folder. Make sure you install the lastest support tools. I used the poolmon /c switch instead of the -c although from what I was reading, they both do the same thing.
Below are the links to what you need to install.
Resource Kit for W2K3
http://www.microsoft.com/downloads/details.aspx?FamilyID=9d467a69-57ff-4ae7-96ee-b18c4790cffd&displaylang=en
Support Tools for W2K3 SP1
http://www.petri.co.il/download_windows_2003_sp1_support_tools.htm
Debugging Tools for W2K3
http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx
Hope this helps,
RadAnonymous
March 17, 2006
Try this one more time as previous two post didn't go
Mathais
The short of it is that you need to install these three things
Resource Kit for W2K3
http://www.microsoft.com/downloads/details.aspx?FamilyID=9d467a69-57ff-4ae7-96ee-b18c4790cffd&displaylang=en
Support Tools for W2K3 SP1
http://www.petri.co.il/download_windows_2003_sp1_support_tools.htm
Debugging Tools for W2K3
http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx
I copied the msdis130.dll and pooltag.txt from the resource kit folder to the support tools folder and was then able to run poolmon /c (From what I can tell it does the same as the -c, new documentation says to use /c)
I did a search on Poolmon as well and couldn't find those files easily. Unfortunately none of the articles tell you where to get it.
Hope this helps,
RadAnonymous
March 17, 2006
Rad - Sorry - it seems like the blog comment filter decided that your comments were spam and unapproved them. Must be the presence of hyperlinks... I've just gone and approved them.
FYI - It's nothing against you or anyone else and I'm sure you mean well, but I personally do not recommend locating and installing arbitrary binaries from arbitrary locations. I continue to be amazed at the arbitrary binaries that a total stranger can influence others to download and run...
//DavidAnonymous
March 17, 2006
Thanks David, The Petro.co.il links straight to microsoft, I just didn't take the time to jot the link down but yes I agree with you about the arbitrary locations.
Here is the actual page that it links to from Microsoft for the Support Tools for W2K3 SP1
http://download.microsoft.com/download/3/e/4/3e438f5e-24ef-4637-abd1-981341d349c7/WindowsServer2003-KB892777-SupportTools-x86-ENU.exeAnonymous
March 19, 2006
The comment has been removedAnonymous
March 21, 2006
Mathias,
Sorry to hear that it didn't show up. I've only had this problem on W2K3 servers but as you could read from about mine wasn't with the even tag. If it didn't appear in the localtag.txt then more than likely it is a third party driver. I remember reading in one of the articles on poolmon to do a search on .sys to find the driver associated with the tag (Just found the link again http://support.microsoft.com/kb/298102/EN-US/ )
After that do a search in google on the driver.
Hope this help.Anonymous
March 27, 2006
Hi everybody.
Thanks for your help with this Problem. It seems, as if the Bitdefender Management Console did the depleting of the memory pool.
I restartet the server (after deinstalling it) and everything seems ok.
The identification of the third-party-driver didn't help in this case.
Thank you for your suggestions and help!
regards,
MathiasAnonymous
March 27, 2006
I need help with this too. Is it just the Nonp entries that can cause and are relevent to this error? Here are my top 10 entries from poolmon text file.
UlHT Paged 1 0 1 8392704 8392704
CM35 Paged 7909 7877 32 4694016 146688
MmCm Nonp 659 4 655 4547104 6942
Obtb Paged 75778 74647 1131 4271136 3776
MmSt Paged 4421779 4419804 1975 4048128 2049
LSwi Nonp 1 0 1 2584576 2584576
TCPt Nonp 1165659 1165632 27 1405616 52059
CMAl Paged 2750 2437 313 1282048 4096
CM25 Paged 1184 965 219 1134592 5180
VoSm Nonp 5832 5812 20 1122480 56124
I hope someone can help me with this or let me know if more information is needed. Thanks in advance.Anonymous
March 27, 2006
Wait a second, I didn't include the very first entry thinking it was a header but looking at it again it must be a valid entry. Here it is:
File Nonp 103274384 102809238 465146 70707328 152
Coule this be my culprit?Anonymous
March 27, 2006
Mathias - The key is to get you to look for something else running on the system that is depleting non-paged-pool. HTTP.SYS is just the messenger; don't shoot. :-)
It's not always obvious what the culprit is, especially the low-level-hooking ones that want to obscure their existence...
//DavidAnonymous
March 27, 2006
Courtney - That's the main culprit (File). It's chewing up over 15x more NonPagedPool Memory than the next user. I don't have it on my system. It did show up on Ryan's poolmon listing above. Perhaps you can diff your software with his and see what is is similar.
Or use poolmon -c to try and find the driver file associated with the "File" tag - which gives hints on what product is causing this.
//DavidAnonymous
March 29, 2006
Hopfully Ryan will stop by and give us an update. I'll list the software that I have on both my servers:
Windows 2003 Server Std edition
Backup Exec 10d (on second server but it backs up first server via File Open Agent)
Symantec 10.0 corporate edition (however I'm upgrading it to 10.1 as I write)
I can't think of anything else that is similar between the servers and put a File tag on both. If my update of Symantec doesn't fix the issue I'm going to set up a lab and recreate my servers. I can't think of another way to do it.Anonymous
March 30, 2006
Hello all,
1. Sorry, I actually went to work for another company and am no longer involved in managign that server. I did run it for two weeks after the NAV upgrade and I did not notice the "leak" any more. I left a message for a previous colleague of mine. When I hear back I will give an update.
2. As far as file is concerned, I was under the impression it had to do with having files open (the meta information on the OS for managing files ect.) . Is there a particular application that is working with lots of files, maybe not closing them correctly? Although, I am not sure to what degree 70,707,328 (70 Meg) is bad.
3. Another thing that affects non paged pool size is the use of the /3gb switch. My first thoughts way back was that the maximum non-paged pool size would increase. I was wrong. If you use the /3GB switch, the maximum non-paged pool size decreases from 256MB to 128MB.
4. Courtney, a listing of installed sofware on you server would probably be helpful. And, I think we all are assuming the erro message you get after some period of time is: "The server was unable to allocate from the system nonpaged pool because the pool was empty."
Ryan M. HagerAnonymous
March 30, 2006
Thanks for posting back Ryan.
I am using the /3GB switch on both servers. A while back I read that it was required for servers using more than 2gb or memory. I too would think having more than 2gb would INcrease the non-paged pool. Looks like a design flaw to me.
I'm going to test your file tag theory that it is related to open files. We program our own reports using Clarion and maybe there's a newly created report that has gone heywire.
To answer number 4, The only error that I noticed was the Connections_Refused error in the HTTP logs. Restarting the server fixed that issue and then I found this website.
I've upgraded Symantec Corporate edition from 10.0 to 10.1. I will keep updating this website on my progress.Anonymous
March 30, 2006
Courtney - Before you call it a "design flaw", you need to understand how Memory Management works on Windows.
By default, a 32bit machine can address 4GB of memory per process. 2GB address space is available for use by User mode applications and the other 2GB is reserved for the Windows Kernel.
Please distinguish between address space and memory allocation. The 2GB is simply available address space to make memory allocations from but are not necessarily all used. Address space is necessary for the kernel to work, as well as efficiently memory map resources between processes, etc (if the kernel can assure that its code maps to the same memory address in all processes, it can do some optimizations while context-switching between processes and NOT remap itself all the time).
/3GB tells Windows to give 3GB address space to User mode and only 1GB for Windows kernel in every process. It is the cheesy way to allow more User mode address space before the existence of 64bit OS which gives far larger address space (and Memory Management is different as a result).
Non Paged Pool memory is special kernel-mode memory and gets its address space from the Windows kernel portion of the memory address. Since /3GB halves the available kernel memory address space, it only makes sense that max NPP size decreases from 256MB to 128MB.
It doesn't make sense for NPP to use user-mode memory address, nor does it make sense for NPP to stay at 256MB at the expense of some other kernel mode memory need.
In other words, /3GB constricts kernel memory address space and hence NPP address space, so seeing it decrease from 256MB to 128MB with /3GB is "by-design". I see no flaw here.
//DavidAnonymous
March 31, 2006
HI Dave, first I would like to say this has site has been a tremendous help in my troubleshooting of IIS6 connection refused problem. This was the only location that gave me a clue about what my IIS6 connectin issue was.
Now I just have to figure out what driver or app is causing the problem. I've reviewed my Poolmon records and it appears that a the DDK drivers are the problem. How do I determine which one?
Here is a sample copy of my poolmon log:
Memory: 2096668K Avail: 1460508K PageFlts: 487 InRam Krnl: 3168K P:92976K
Commit:2418032K Limit:8345212K Peak:2472812K Pool N:49240K P:94076K
System pool information
Tag Type Allocs Frees Diff Bytes Per Alloc
Ddk Nonp 487036 ( 42) 429814 ( 36) 57222 21111424 ( 2016) 368
File Nonp 4611529 ( 211) 4564638 ( 208) 46891 7132072 ( 456) 152
LSwi Nonp 1 ( 0) 0 ( 0) 1 2576384 ( 0) 2576384
TCPc Nonp 156911 ( 12) 116183 ( 9) 40728 1954944 ( 144) 48
TCPt Nonp 246614 ( 2) 246587 ( 2) 27 1392560 ( 0) 51576
Irp Nonp 200406 ( 0) 197990 ( 1) 2416 1015800 ( -448) 420
VickydAnonymous
April 05, 2006
Hello all,
Ok, I did get an update from my former colleague . "tdll" is still an issue for them. They are going to rebuild on SVR 2003 and see if the problem persists.
Ryan M. HagerAnonymous
April 07, 2006
I un-installed a CommView and that stopped all leaks. The driver for this application was the problem.
VickydAnonymous
May 11, 2006
I was wondering if someone could help me figure this one out. I have a W2k3 SP1 server that's refusing connections every 3 - 4 weeks. It appears that Afdb is the tag that is growing. Here are the poolmon captures:
Memory: 2096400K Avail: 796916K PageFlts: 228 InRam Krnl: 3704K P:89264K
Commit:1095580K Limit:4043356K Peak:1257404K Pool N:142776K P:91028
System pool information
Tag Type Allocs Frees Diff Bytes Per Alloc
AfdB Nonp 265443 ( 0) 59297 ( 0) 206146 103506360 ( 0) 50
R100 Nonp 21 ( 0) 0 ( 0) 21 9585384 ( 0) 456446
MmCm Nonp 585 ( 0) 32 ( 0) 553 3326496 ( 0) 6015
tdLL Nonp 27230 ( 0) 21771 ( 0) 5459 2998680 ( 0) 549
File Nonp 52425165 ( 65) 52408019 ( 65) 17146 2692304 ( 0) 157
LSwi Nonp 1 ( 0) 0 ( 0) 1 2584576 ( 0) 2584576
TCPt Nonp 699513 ( 5) 699485 ( 5) 28 1456216 ( 0) 52007
PTrk Nonp 655930 ( 0) 651945 ( 0) 3985 1196840 ( 0) 300
Ntfr Nonp 1062944 ( 0) 1045950 ( 0) 16994 1088584 ( 0) 64
MmCa Nonp 3450887 ( 13) 3442105 ( 13) 8782 974976 ( 0) 111
Thre Nonp 2981123 ( 39) 2979785 ( 39) 1338 834912 ( 0) 624
TCPA Nonp 40121 ( 0) 38423 ( 0) 1698 624864 ( 0) 368
Pool Nonp 6 ( 0) 3 ( 0) 3 610304 ( 0) 203434
AfdE Nonp 40618 ( 0) 38859 ( 0) 1759 492520 ( 0) 280
NtFs Nonp 1819450 ( 0) 1808907 ( 0) 10543 425464 ( 0) 40
Irp Nonp 945247 ( 2) 944182 ( 0) 1065 423592 ( 960) 397
Ntfn Nonp 652134 ( 0) 641592 ( 0) 10542 422864 ( 0) 40
LSwr Nonp 128 ( 0) 0 ( 0) 128 416768 ( 0) 3256
CcSc Nonp 1958366 ( 0) 1957168 ( 2) 1198 373776 ( -624) 312
Devi Nonp 1688 ( 0) 1320 ( 0) 368 366368 ( 0) 995
Memory: 2096400K Avail: 657416K PageFlts: 330 InRam Krnl: 3704K P:81644K
Commit:1187564K Limit:4043356K Peak:1257404K Pool N:196036K P:83344
System pool information
Tag Type Allocs Frees Diff Bytes Per Alloc
AfdB Nonp 394396 ( 0) 86013 ( 0) 308383 155097912 ( 0) 50
R100 Nonp 21 ( 0) 0 ( 0) 21 9585384 ( 0) 456446
MmCm Nonp 585 ( 0) 32 ( 0) 553 3326496 ( 0) 6015
tdLL Nonp 33103 ( 0) 27643 ( 0) 5460 2999232 ( 0) 549
LSwi Nonp 1 ( 0) 0 ( 0) 1 2584576 ( 0) 2584576
File Nonp 68066779 ( 90) 68052843 ( 91) 13936 2145600 ( -152) 153
TCPt Nonp 894609 ( 11) 894581 ( 11) 28 1456216 ( 0) 52007
PTrk Nonp 863427 ( 0) 859206 ( 0) 4221 1267176 ( 0) 300
TCPA Nonp 52932 ( 0) 50457 ( 0) 2475 910800 ( 0) 368
Thre Nonp 3799300 ( 49) 3797879 ( 47) 1421 886704 ( 1248) 624
AfdE Nonp 53618 ( 0) 51082 ( 0) 2536 710080 ( 0) 280
Pool Nonp 6 ( 0) 3 ( 0) 3 610304 ( 0) 203434
Ntfr Nonp 1373312 ( 0) 1365233 ( 0) 8079 518024 ( 0) 64
MmCa Nonp 4489055 ( 13) 4485021 ( 14) 4034 442624 ( -96) 109
Irp Nonp 1020062 ( 2) 1018966 ( 0) 1096 435368 ( 960) 397
LSwr Nonp 128 ( 0) 0 ( 0) 128 416768 ( 0) 3256
Devi Nonp 2026 ( 0) 1658 ( 0) 368 366368 ( 0) 995
Even Nonp 9250970 ( 151) 9243598 ( 151) 7372 360000 ( 0) 48
Vad Nonp 5896241 ( 22) 5890766 ( 27) 5475 262800 ( -240) 48
CcSc Nonp 2518751 ( 2) 2517918 ( 2) 833 259896 ( 0) 312
MmCi Nonp 6732 ( 0) 5605 ( 0) 1127 253392 ( 0) 224
TCPc Nonp 349953 ( 0) 344732 ( 0) 5221 250608 ( 0) 48
I found an article that says that there was a problem with afd.sys that was supposed to be fixed in Win2000 SP3. Here's the article:
http://support.microsoft.com/kb/296265
Anyone have any ideas what might be causing this?
Thanks,
MarkAnonymous
May 15, 2006
Mark,
What software are you using? More specifically, assuming AFD=Winsock like the article suggests, what apps are you running on this machine that are communicating over TCP-IP? I am not sure if it matters, but are you using a NIC that does TCP Offloading?
Ryan.Anonymous
May 17, 2006
I followed all the great info here, and found that the NTFC or NTFS driver was taking up about 60MB. After restarting, it's down to almost nothing. What could cause this?Anonymous
May 18, 2006
Ryan,
Thanks for the response. It has SQL Server 2000, IIS, SMTP Service, Norton AntiVirus, Windows Services for Unix and a USQL client app for connecting to a Cobol db. The NIC does TCP Offloading. The server is also used to print custom labels for use in the business and I'm starting to think that printing is the problem. It's hard to find a time when there isn't any traffic to the server but it seems that if I watch Poolmon, the Afdb tag grows every time I print a label.
MarkAnonymous
May 18, 2006
Mark,
I am not sure how you are printing, and that could be it. However, my first thought was the USQL client. Does your printing application also use the USQL client to connect to the DB? It may be a combination of Label Printing Application and USQL client. Additionally, are you using any special print queue software?
Ryan M. HagerAnonymous
May 24, 2006
Hi,
We were having this exact problem on a few of our Windows 2003 IIS 6 machines. A search led me to this blog entry. While the information provided is extremely helpful, we have hit a wall with finding the actual culprit.
As a few others have reported here and elsewhere on the web, something that uses the tag "File" is consuming an ever increasing amount of NPP memory. I have yet to determine which driver this is. Has anyone else had any luck with this? Using the findstr command I find 76 drivers. I can post that list here if anyone thinks it would be helpful.
Here is some output from poolmon -b -c -p, including the top five users of NPP:
Memory: 2096580K Avail: 700456K PageFlts: 316 InRam Krnl: 2192K P:80572K
Commit:1458792K Limit:4042240K Peak:1552600K Pool N:200948K P:81972K
System pool information
Tag Type Allocs Frees Diff Bytes Per Alloc Mapped_Driver
File Nonp 9946333 ( 193) 9149316 ( 161) 797017 121175032 ( 4864) 152 Unknown Driver
TCPc Nonp 1595850 ( 66) 810930 ( 34) 784920 37676160 ( 1536) 48 [tcpip]
NDam Nonp 224898 ( 2) 223669 ( 4) 1229 4192952 (-32768) 3411 [ndis]
MmCm Nonp 546 ( 0) 4 ( 0) 542 3891744 ( 0) 7180 Unknown Driver
LSwi Nonp 1 ( 0) 0 ( 0) 1 2584576 ( 0) 2584576 Unknown Driver
The two third party applications that are running at the time those numbers were taken are SmarterStats 3 and Visnetic Firewall. Stopping both services shows no noticeable change in the NPP usage.
Any suggestions or comments would be greatly appreciated. If any further information is required, please let me know. Thank you.Anonymous
May 25, 2006
The comment has been removedAnonymous
May 25, 2006
The comment has been removedAnonymous
May 26, 2006
The comment has been removedAnonymous
May 31, 2006
Sorry it took me so long to respond.
We've recently gotten in contact with VisNetic regarding the firewall. They have acknowleged that the product can cause these issues on Windows 2003 and have offered a beta replacement for the ambrim.sys driver. I'll post back with my findings.
-MikeAnonymous
June 08, 2006
I just wanted to post a followup to my previous posts. I ended up calling Microsoft and we setup poolmon and perfmon to constantly run and capture data. After a couple of days I sent them the logs and they found that the leak was in the HP print driver. We updated the HP print driver to the latest version for the HP4250 printers we have and also changed the setup of the printers to use a Standard TCP/IP port rather than the HP Standard TCP/IP port. Also, just to be on the safe side they had me run an update for Norton VirusScan. After a couple of weeks it appears that the leak has been eliminated.
Thanks for the help you all have provided!
MarkAnonymous
June 09, 2006
Another update for our issues. The replacement ambrim.sys driver provided by VisNetic had no positive effects. I've disabled the HTTP filtering feature of the firewall and that seems to have alleviated the NPP memory usage problems. Perhaps I should approach 8Signs/C&C Software directly about this and leave VisNetic out of it.
-MikeAnonymous
June 11, 2006
I'm having a similar problem with running out of non-paged memory. In my case, the tag that is responsible is "Devi" - can anyone suggest a way to work out which specific driver would be responsbile?
Memory: 2096616K Avail: 1488572K PageFlts: 403 InRam Krnl: 3928K P:65404K
Commit: 460988K Limit:4034104K Peak: 498364K Pool N:78908K P:71948K
Tag Type Allocs Frees Diff Bytes Per Alloc
Devi Nonp 14065 ( 40) 540 ( 0) 13525 47746248 (145280) 3530
Wdm Nonp 257238 ( 807) 256463 ( 807) 775 6189872 ( 0) 7986
NVP Nonp 2010 ( 0) 0 ( 0) 2010 2888320 ( 0) 1436
ISil Nonp 7185 ( 4) 7165 ( 4) 20 1638096 ( 0) 81904
MmCm Nonp 706 ( 0) 64 ( 0) 642 1362952 ( 0) 2122
Stor Nonp 3076 ( 0) 1083 ( 0) 1993 1326576 ( 0) 665
tdLL Nonp 2190 ( 0) 354 ( 0) 1836 1211600 ( 0) 659
File Nonp 312933 ( 295) 306434 ( 295) 6499 1032536 ( 0) 158
SNPU Nonp 465 ( 0) 1 ( 0) 464 1027656 ( 0) 2214
NV Nonp 2748 ( 0) 1876 ( 0) 872 827616 ( 0) 949
NRAT Nonp 2 ( 0) 0 ( 0) 2 516096 ( 0) 258048
Thanks,
Nick PriceAnonymous
June 11, 2006
Nick - Since you want to troubleshoot your situation, I suggest carefully reading both this blog entry as well as the associated comments since they both have a lot of information on how to figure out what you want.
For example, the blog entry mentions:
http://support.microsoft.com/?id=177415
Which also mentions:
http://support.microsoft.com/?id=298102
//DavidAnonymous
June 12, 2006
excellent. just what i needed.Anonymous
July 04, 2006
Hi all,
I have the exact problem on a Windows 2003 IIS 6 machines. This information in this blog is very helpfull. With the help of poolmon I figure out the problem on our server. The Watchguard Admin Gui causes the memory problem.
Thanks to all
schobiAnonymous
July 23, 2006
I seem to be experiencing the same problem with two of my servers.
Server A has "tdLL" at the top of the list, at 99005465 bytes, and second on the list is "SavE" at 16529752 bytes, at the time of writing.
Server B, however, has "SavE" at the top of its list with the same bytes as Server A, and with "tdLL" nowhere to be found.
Both servers are already running Symantec AntiVirus ver 10.0.0.359. The only difference between them is that Server B is running the AntiVirus as a Server instead of a client.
So I'm not really sure if "tdLL" belongs to Symantec, and it's not a good idea to uninstall the Antivirus from Server A to find out because I don't have another authorised antivirus software to protect Server A.
Any suggestion to my next step?
JK.Anonymous
July 23, 2006
Jessie Kom - people have reported tdLL leaks going away with Symantec AV 10, though unclear what patch/sub-build number. You do not appear to have the latest at 10.1 nor 10.1.
I suggest contacting Symantec support regarding the issue and getting their latest AV 10 updates. Since you need to run that software, I do not recommend trying to uninstall it from an investigation point-of-view.
Basically, there is no way for you to deal with the memory leak other than to either:
1. NOT run the code with the leak
2. run code without the leak
And since you cannot do #1, you must do #2 - meaning you must contact and obtain the latest patch, and if it does not work, remain with Symantec support until they prove/show that tdLL is not theirs or is not their issue.
//DavidAnonymous
July 25, 2006
Thanks David.
I've just downloaded 10.0.2 from Symantec, it's not the latest patch but that's all I can download, probably due to the support plan my company got.
However, yeah, I'll be checking with the Symantec support on this "tdLL" and the patches I can get.
Will update you guys again...Anonymous
July 28, 2006
The comment has been removedAnonymous
July 31, 2006
The comment has been removedAnonymous
August 17, 2006
David,
Great article. its the most insightful I've seen on the topic in my few days of searching.
My company is loadtesting a new asp.net application running on windows 2003 SP1, IIS 6, 2 gig ram and 2 cpu's. We are loadtesting using mercury loadrunner. When the load gets to about 140 users loadrunner starts receiving 100061 Connection Refused messages. the strange thing is I don't see any Connection_Refused messages in the httperr#.log. The processors are only about 50 % used when this error begins.
Any idea's? I don't think it is another process using up memory; instead I think it might be ASP.net or IIS running out of connections or threads but haven't found any way of verifying this.
Thanks,
LouAnonymous
August 17, 2006
The comment has been removedAnonymous
September 01, 2006
Thanks for the info in this Blog. I have got the problem with the File tag. Trying to track it down. Will let you know what I find.
Memory: 2096572K Avail: 127812K PageFlts: 28 InRam Krnl: 2112K P:24936K
Commit:2116980K Limit:4042232K Peak:2237084K Pool N:246824K P:46552K
Tag Type Allocs Frees Diff Bytes Per Alloc
File Nonp 1432693511 ( 842) 1431507006 ( 847) 1186505 180352184 ( -776) 1
Thre Nonp 418801034 ( 214) 418795362 ( 218) 5672 3539328 ( -2496) 624
LSwi Nonp 1 ( 0) 0 ( 0) 1 2584576 ( 0) 2584576
MmCm Nonp 42 ( 0) 16 ( 0) 26 2015776 ( 0) 77529
TCPt Nonp 21858688 ( 7) 21858659 ( 7) 29 1456480 ( 0) 50223Anonymous
September 04, 2006
I have had three live servers with the problem. All with the File tag at around 200MB. I had to reboot those servers to get the users back on, but I have now found a dev server that is not being used and is currently at 194MB and I am getting the Connection Refused error.
I am having problems trying to track down the File tag. As with some of the other entries here, the word File appears in a number of drivers. I have also tried hFile as recommended in one of the Microsoft articles and I come up with en IBM TSM driver. However, I have another PC that has the File tag, but when I search for hFile, it does not return anything....so I am not sure I trust the IBM find, although it is file related.....still searching.
RgdsAnonymous
September 04, 2006
I have found the problem.
It was an IBM WMI service with a process called PegasusProviderAdapter.exe. Killed this off (as it did not matter what services I stopped...it didn't) and the File tag went back to 400K.
It was also highlighted by using the Task Manager and showing the Handles column. This was over 1 million. Also the FileMon tool from SysInternals showed a lot of access from the exe.
So, if you problem is the File tag, check Task manager for Handles.
Thanks all above for your pointers....it got me there.Anonymous
September 05, 2006
Really helpful, but what do we do when:
poolmon /c
gives us the following output
Poolmon: No localtag.txt in current directory
Poolmon: Unable to load required dlls, cannot create local tag file
I gather that this means we can't generate a localtag.txt file and therefore can't workout which driver is the problem - any ideas?Anonymous
September 15, 2006
Hello
Finally i found the right place. Great postings.
My problem is with clustered active/passive exchange 2003 running on windows2003 ent. Both nodes have 4GB of RAM and Boot.ini includes /3GB and USERVA=3030 switches. When i check my NPP Pool is actually shows around 250Megs instead of the ususal 128Megs
I too keep getting connections refused after few days
It manifest itself within my HTTP cluster resource which goes off line and then it simply dies. The only way to solve this problem is to fail over to the other node. This system was build about 2 years ago and only after applying service pack 2 for exchange 2003 i;m seeing this error.
My top 6 main culprits from poolmon
Memory: 3669384K Avail: 1157048K PageFlts: 89870 InRam Krnl: 2812K P:145184K
Commit:1974712K Limit:7718392K Peak:2047588K Pool N:96468K P:146244K
System pool information
Tag Type Allocs Frees Diff Bytes Per Alloc
MmCm Nonp 5861 ( 0) 202 ( 0) 5659 25068320 ( 0) 4429
File Nonp 24187515 (4013) 24127756 (4050) 59759 9096488 ( -5672) 152
TCPT Nonp 64297 ( 13) 44875 ( 20) 19422 7924608 ( -2856) 408
AfdE Nonp 801580 ( 332) 781989 ( 339) 19591 5485480 ( -1960) 280
Irp Nonp 137215759 (15216) 137204307 (15253) 11452 4891440 ( -9056) 427
Mdl Nonp 287772 ( 4) 273441 ( 4) 14331 4850408 ( 0) 338Anonymous
September 19, 2006
The comment has been removedAnonymous
October 02, 2006
Hi guys, I'm so sorry I actually forgot to update you! My problem was confirmed resolved after I upgraded the SAV server and clients on the servers to version 10.0.2.2021. The "tdLL" tag on one server running SAV client is now gone for good, and a leaking "NtFC" on another server running SAV server is also gone. It's been a few couple of months now, and my monitoring still shows me stable NPP usage. Cheers!Anonymous
April 21, 2007
Hi, We have examined this document. we are receiving same error which is specified in this document and we have also install this poolmoon.exe but, we could not find how to resolve this issue in this document so, please add some resolution document on this error. Thanks, Mehul Choksi.Anonymous
May 02, 2007
The error of connection refused, leaves IIS unresponsive.... below is the result of poolmon.exe, if someone can advice on it. Memory: 4062672K Avail: 2774072K PageFlts: 37671 InRam Krnl: 2500K P:233456K Commit:1211472K Limit:6000160K Peak:1216708K Pool N:169772K P:235880 System pool information Tag Type Allocs Frees Diff Bytes Per Alloc File Nonp 5116787 (3619) 4559778 (3183) 557009 85373928 ( 66272) 153 TCPc Nonp 959614 ( 572) 482062 ( 286) 477552 22922496 ( 13728) 48 MmCa Nonp 450958 ( 475) 374788 ( 401) 76170 8523088 ( 7984) 111 Ntfr Nonp 126776 ( 266) 10695 ( 0) 116081 7430152 ( 17024) 64 NDam Nonp 57933 ( 41) 56351 ( 42) 1582 7365312 (-16384) 4655 CcSc Nonp 285330 ( 272) 270347 ( 239) 14983 4794560 ( 10560) 320 MmCm Nonp 663 ( 0) 4 ( 0) 659 4456992 ( 0) 6763 Ntfn Nonp 111696 ( 156) 27099 ( 48) 84597 3385360 ( 4320) 40 NtFs Nonp 381250 ( 615) 296715 ( 570) 84535 3382648 ( 1800) 40 LSwi Nonp 1 ( 0) 0 ( 0) 1 2584576 ( 0) 2584576 Irp Nonp 698615 ( 0) 694779 ( 1) 3836 1668008 ( -160) 434 TCPt Nonp 1868090 (1366) 1867946 (1366) 144 1471912 ( 0) 10221 TPLA Nonp 256 ( 0) 0 ( 0) 256 1048576 ( 0) 4096 Thre Nonp 55309 ( 46) 54022 ( 23) 1287 803088 ( 14352) 624 Wmit Nonp 13 ( 0) 0 ( 0) 13 667648 ( 0) 51357 Pool Nonp 6 ( 0) 3 ( 0) 3 610304 ( 0) 203434 FSfm Nonp 87889 ( 111) 74653 ( 9) 13236 529440 ( 4080) 40 ReTa Nonp 13744 ( 53) 887 ( 0) 12857 516360 ( 2120) 40 Io Nonp 4471379 (3014) 4470604 (3017) 775 495528 ( 8032) 639 ReEv Nonp 22072 ( 13) 2210 ( 0) 19862 476688 ( 312) 24 Ddk Nonp 3118 ( 0) 460 ( 0) 2658 425280 ( 0) 160 LSwr Nonp 128 ( 0) 0 ( 0) 128 416768 ( 0) 3256 Even Nonp 517899 ( 298) 510041 ( 173) 7858 380368 ( 6016) 48 UlCO Nonp 2624 ( 0) 2000 ( 1) 624 379392 ( -608) 608 Stat Nonp 505305 ( 305) 503049 ( 298) 2256 379008 ( 1176) 168 Devi Nonp 446 ( 0) 116 ( 0) 330 356280 ( 0) 1079 MmCi Nonp 1582 ( 0) 140 ( 0) 1442 313888 ( 0) 217 UlCJ Nonp 1579 ( 0) 0 ( 0) 1579 290536 ( 0) 184 Mm Nonp 90 ( 0) 72 ( 0) 18 284536 ( 0) 15807 Vad Nonp 908965 ( 701) 903490 ( 483) 5475 262800 ( 10464) 48 TCPp Nonp 64 ( 0) 0 ( 0) 64 262144 ( 0) 4096 AfdC Nonp 467791 ( 283) 466222 ( 283) 1569 251040 ( 0) 160 NDpp Nonp 74 ( 0) 1 ( 0) 73 250816 ( 0) 3435 Mdl Nonp 5472 ( 0) 3565 ( 0) 1907 248136 ( 0) 130 TCPB Nonp 194439 ( 93) 194300 ( 93) 139 214696 ( 0) 1544Anonymous
May 22, 2007
I get IIS crashing about every 2-3 hours. When i run poolmon it lists the tag mmcm using almost 60MB or non paged ram. Does anyone know what this is and how to fix it.Anonymous
June 18, 2007
The comment has been removedAnonymous
September 24, 2007
Vickyd, Were you able to find what driver was associated to the Ddk tag? Can someone please help find what could cause the Ddk driver to take the most memory? Thank you, Hiram Here is a sample copy of my poolmon log: Tag Type Allocs Frees Diff Bytes Per Alloc Ddk Nonp 4247411 2134379 21 13032 1 86928776 88 MmCm Nonp 31471 30363 1108 1 3652784 12322 Thre Nonp 29212 27806 1406 877344 624 WLBS Nonp 36050 36038 12 766528 63877 R100 Nonp 28 2 26 9 658968 371498 File Nonp 1183112 1179027 4085 622344 152 Pool Nonp 6 3 3 610304 203434Anonymous
October 18, 2007
I am almost having a problem with a Ddk driver, but cannot figure out what one. The methods listed in KB298102 to find what drivers are using the Ddk tag does not seem to work.Anonymous
November 05, 2007
I am having this issue but it is the MmCm continually growing until connection refused shows up in the logs. Here is the poolmon read from a Server 2003 SP2. Memory: 8386228K Avail: 7580940K PageFlts: 659 InRam Krnl: 3472K P:60848K Commit: 580648K Limit:10281340K Peak: 593208K Pool N:240764K P:61784 System pool information Tag Type Allocs Frees Diff Bytes Per Alloc NTID Nonp 3068013 ( 86) 2807570 ( 76) 260443 205441816 ( 960) 788 MmCm Nonp 241685 ( 1) 241524 ( 1) 161 9434384 ( 0) 58598 UlHT Paged 1 ( 0) 0 ( 0) 1 4198400 ( 0) 4198400 NDpp Nonp 987 ( 0) 0 ( 0) 987 3930304 ( 0) 3982 Wdm Nonp 2 ( 0) 0 ( 0) 2 3420160 ( 0) 1710080 LSwi Nonp 1 ( 0) 0 ( 0) 1 2576384 ( 0) 2576384 MmSt Paged 182187 ( 7) 180907 ( 8) 1280 2213072 ( -520) 1728 TCPt Nonp 399304 ( 11) 399274 ( 11) 30 1458080 ( 0) 48602 TPLA Nonp 256 ( 0) 0 ( 0) 256 1048576 ( 0) 4096 Mdl Nonp 184477 ( 38) 178670 ( 36) 5807 857824 ( 256) 147 brcm Nonp 104343 ( 0) 104319 ( 0) 24 851968 ( 0) 35498 Gh15 Paged 2809 ( 16) 2709 ( 16) 100 808528 ( 0) 8085 File Nonp 7593995 ( 156) 7589981 ( 158) 4014 612384 ( -304) 152 Thre Nonp 160179 ( 3) 159259 ( 11) 920 574080 ( -4992) 624 CM16 Paged 501 ( 0) 372 ( 0) 129 557056 ( 0) 4318 I found tech article 944011 and have update the nic drivers and team drivers and it is still climbing. Any suggestions would be helpful. THis is a IIS6 server with asp and WebSphere Edgeserver components running. MichaelAnonymous
February 20, 2008
Overview It’s not unnatural to assume an IIS process hang when web client browsers begin reporting eitherAnonymous
February 20, 2008
Overview It’s not unnatural to assume an IIS process hang when web client browsers begin reporting eitherAnonymous
February 28, 2008
It may be "natural", but that is not necessarily correct. People frequently confuse what seems natural to them as "their way of thinking is correct", so they assume they are correct... and we all know how far that flies... //DavidAnonymous
February 28, 2008
Corey - your problem does not sound like this issue. It sounds like you are running unstable web applications and should diagnose it using my other blog entries on how to diagnose Application Pool Crashes. //DavidAnonymous
February 28, 2008
paritoshjani - for FILE pooltag, it indicates that there are lots of open file handles in user mode, causing kernel mode depletion of NPP. You should check with Task Manager to see what process(s) have the most "Handles" and stop it -- that will most likely be the culprit which is leaking the file handles in user mode and thus the FILE pooltag in kernel mode. //DavidAnonymous
March 17, 2008
Hi, I'm having the same issue. I'm running an Exchange front-end/back-end config and my back-end IIS site goes down every once in a while because of this problem (connection refused). Here's a screenshot of poolmon when the problem exists: http://img167.imageshack.us/img167/9890/poolmonbeforerebootto4.gif The two largest tags are File (50MB) and MmCm (34Mb). These are relatively small compared to others who have posted, nonetheless, the problem exists. I generated then looked through the localtag.txt file but could not find either of these tags. My server is a DL385G2 so I downloaded the latest PSP pack (contains all drivers for the server) and installed it. We'll see if the problem comes back next week.Anonymous
March 17, 2008
I restarted the system and File takes up only 1.8MB. After about 2 hours, it grew to 2.2MB. Running "findstr /m /l File *.sys" under c:windowssystem32drivers returns a long list of drivers. At this point, I'm stuck. I can track down each and every one of those drivers and try to remove the ones I don't need but that's asking for trouble since I can't be sure which one is safe to remove. Is there a better solution? Somebody posted about an IBM-related driver causing the leak on his system but I don't have that on my system. So far, JohnE seems to have the answer for those with "File" tag problems. Viewing the number of handles using Task Manager when the problem occurs should point you to the culprit. Unfortunately for me, I've already restarted my server so I'll have to wait until the next occurrence.Anonymous
May 15, 2008
Hi, Since the launch of our new W2k3 Webserver, we encounter this connexion_refused problem each week. I've followed the whole tutorial and find some tags that could cause the NPP occupation : . file . even After a findstr in the drivers folders, it seems that the tag "even" is linked to the AMBRAPP.SYS used by my firewall (8 signs). I've patched it in last release and verified that the driver has been correctly patched : it has been done. The connexion_refused just came back. I've patched my NIC drivers and all that could be patched, deleted all softs that were useless, stopped all services that were useless. The connexion_refused just came back. I don't know what to do more. Here is an extract of my poolmon when the connexion_refused came today : Memory: 4192636K Avail: 1756012K PageFlts: 1305 InRam Krnl: 2556K P:52452K Commit:2771316K Limit:6117192K Peak:2817516K Pool N:109652K P:53576K System pool information Tag Type Allocs Frees Diff Bytes Per Alloc File Nonp 12396850 ( 179) 11996683 ( 176) 400167 60826792 ( 392) 152 TCPc Nonp 1329917 ( 12) 932866 ( 7) 397051 19058448 ( 240) 48 NDam Nonp 55692 ( 0) 54550 ( 0) 1142 3207936 ( 0) 2809 NAI0 Nonp 8524200 ( 162) 8523805 ( 158) 395 2969816 ( 1120) 7518 LSwi Nonp 1 ( 0) 0 ( 0) 1 2576384 ( 0) 2576384 MmCm Nonp 72 ( 0) 33 ( 0) 39 1733016 ( 0) 44436 TCPt Nonp 2803864 ( 51) 2803817 ( 51) 47 1459936 ( 0) 31062 Io Nonp 25209203 ( 433) 25208578 ( 438) 625 1308264 (-40960) 2093 TPLA Nonp 256 ( 0) 0 ( 0) 256 1048576 ( 0) 4096 Thre Nonp 211269 ( 0) 209810 ( 5) 1459 910416 ( -3120) 624 Even Nonp 31492660 ( 184) 31476918 ( 195) 15742 856176 ( -528) 54 Irp Nonp 847400 ( 10) 845278 ( 18) 2122 785320 ( -2144) 370 VadS Nonp 20987784 ( 103) 20964716 ( 108) 23068 738176 ( -160) 32 Ddk Nonp 4002 ( 0) 352 ( 0) 3650 583944 ( 0) 159 LSwr Nonp 128 ( 0) 0 ( 0) 128 416768 ( 0) 3256 Mdl Nonp 2533985 ( 100) 2530766 ( 101) 3219 413552 ( -128) 128 Pool Nonp 5 ( 0) 2 ( 0) 3 348160 ( 0) 116053 AfdC Nonp 499106 ( 4) 496991 ( 2) 2115 338400 ( 320) 160 Mm Nonp 440 ( 0) 425 ( 0) 15 306040 ( 0) 20402 Vad Nonp 6256579 ( 24) 6250357 ( 24) 6222 298656 ( 0) 48 NDpp Nonp 162 ( 0) 73 ( 0) 89 292256 ( 0) 3283 Sema Nonp 6016269 ( 24) 6011717 ( 24) 4552 255376 ( 0) 56 Devi Nonp 6014 ( 0) 5681 ( 0) 333 246168 ( 0) 739 TCPp Nonp 59 ( 0) 0 ( 0) 59 241664 ( 0) 4096 Stat Nonp 485622 ( 5) 484248 ( 10) 1374 230832 ( -840) 168 Dump Nonp 6 ( 0) 0 ( 0) 6 229776 ( 0) 38296 TCPC Nonp 67410 ( 0) 64981 ( 0) 2429 206568 ( 0) 85 None Nonp 32 ( 0) 0 ( 0) 32 200448 ( 0) 6264 Hal Nonp 5870290 ( 258) 5870279 ( 258) 11 197616 ( -72) 17965 Ntf0 Nonp 3 ( 0) 0 ( 0) 3 196608 ( 0) 65536 AfdE Nonp 928161 ( 9) 927472 ( 7) 689 192920 ( 560) 280 MmCa Nonp 3702374 ( 34) 3700523 ( 34) 1851 185600 ( 0) 100 Ntfr Nonp 25528 ( 0) 22754 ( 0) 2774 178504 ( 0) 64 brcm Nonp 24 ( 0) 12 ( 0) 12 164544 ( 0) 13712 usbp Nonp 195 ( 0) 145 ( 0) 50 158096 ( 0) 3161 RxHb Nonp 201 ( 0) 0 ( 0) 201 155976 ( 0) 776 MmCi Nonp 3198 ( 0) 2545 ( 0) 653 150432 ( 0) 230 RceT Nonp 1 ( 0) 0 ( 0) 1 131072 ( 0) 131072 CcSc Nonp 1238135 ( 25) 1237777 ( 25) 358 114560 ( 0) 320 UlCO Nonp 1536 ( 0) 1355 ( 0) 181 110048 ( 0) 608 UlLS Nonp 6073 ( 2) 6059 ( 0) 14 107144 ( 16384) 7653 Vadl Nonp 223433 ( 1) 221783 ( 6) 1650 105600 ( -320) 64 SeTd Nonp 18304474 ( 137) 18302931 ( 134) 1543 98752 ( 192) 64 TCPA Nonp 802441 ( 8) 802246 ( 6) 195 71760 ( 736) 368 Ntfi Nonp 85855 ( 1) 85607 ( 9) 248 67456 ( -2176) 272 AmlH Nonp 1 ( 0) 0 ( 0) 1 65536 ( 0) 65536 NtFs Nonp 491804 ( 31) 490239 ( 37) 1565 65096 ( -240) 41 TCPT Nonp 64573 ( 0) 64431 ( 0) 142 64816 ( 0) 456 Ntfn Nonp 312607 ( 25) 311027 ( 31) 1580 63792 ( -240) 40 Bfpl Nonp 40 ( 0) 0 ( 0) 40 62400 ( 0) 1560 Info Nonp 114430 ( 3) 114429 ( 3) 1 61440 ( 0) 61440 Ica Nonp 14278 ( 38) 14208 ( 37) 70 60672 ( 56) 866 ReTa Nonp 16290 ( 2) 15062 ( 0) 1228 53696 ( 80) 43 IpSI Nonp 1027 ( 0) 0 ( 0) 1027 53272 ( 0) 51 Fltr Nonp 1100 ( 0) 0 ( 0) 1100 52800 ( 0) 48 FSfm Nonp 250792 ( 8) 249563 ( 34) 1229 49160 ( -1040) 40 UlFU Nonp 56802 ( 0) 56788 ( 0) 14 46200 ( 0) 3300 AfdB Nonp 160933 ( 0) 160873 ( 0) 60 44464 ( 0) 741 Port Nonp 3335726 ( 50) 3334424 ( 47) 1302 41664 ( 96) 32 CcVa Nonp 1 ( 0) 0 ( 0) 1 40960 ( 0) 40960 Muta Nonp 36305 ( 0) 35770 ( 0) 535 39328 ( 0) 73 CMpa Nonp 13280 ( 2) 12686 ( 1) 594 38016 ( 64) 64 Proc Nonp 350 ( 0) 294 ( 0) 56 37184 ( 0) 664 Dnod Nonp 291 ( 0) 168 ( 0) 123 36408 ( 0) 296 NtFL Nonp 204599 ( 5) 204591 ( 5) 8 34768 ( 0) 4346 PooL Nonp 4 ( 0) 0 ( 0) 4 32768 ( 0) 8192 AmlC Nonp 72 ( 0) 68 ( 0) 4 32768 ( 0) 8192 UlOT Nonp 8 ( 0) 0 ( 0) 8 32768 ( 0) 4096 VoSb Nonp 8601 ( 0) 8599 ( 0) 2 32768 ( 0) 16384 Lfsr Nonp 2 ( 0) 0 ( 0) 2 32768 ( 0) 16384 WmiG Nonp 1192 ( 0) 1054 ( 0) 138 30912 ( 0) 224 RxNr Nonp 2537 ( 0) 2530 ( 0) 7 26888 ( 0) 3841 LScn Nonp 72 ( 0) 35 ( 0) 37 26344 ( 0) 712 Driv Nonp 112 ( 0) 11 ( 0) 101 25392 ( 0) 251 CPnp Nonp 10896 ( 0) 10799 ( 0) 97 24832 ( 0) 256 Ioin Nonp 16 ( 0) 5 ( 0) 11 22880 ( 0) 2080 ReEv Nonp 16677 ( 1) 15786 ( 2) 891 21384 ( -24) 24 Qphf Nonp 236 ( 0) 232 ( 0) 4 21336 ( 0) 5334 Evnt Nonp 20 ( 0) 0 ( 0) 20 21120 ( 0) 1056 SMBi Nonp 105 ( 0) 91 ( 0) 14 20752 ( 0) 1482 TCPB Nonp 14187 ( 0) 14173 ( 0) 14 20696 ( 0) 1478 AcpD Nonp 187 ( 0) 128 ( 0) 59 20120 ( 0) 341 ReSe Nonp 17447 ( 1) 16828 ( 0) 619 19808 ( 32) 32 UlIR Nonp 48743 ( 1) 48730 ( 1) 13 19560 ( 0) 1504 CM44 Nonp 13284 ( 2) 12687 ( 1) 597 19080 ( 32) 31 Time Nonp 1482 ( 0) 1379 ( 0) 103 19000 ( 0) 184 CcVl Nonp 90 ( 0) 64 ( 0) 26 18336 ( 0) 705 UlHR Nonp 3774 ( 0) 3760 ( 0) 14 17808 ( 0) 1272 RxNf Nonp 35235 ( 0) 35140 ( 0) 95 16808 ( 0) 176 MmPd Nonp 1 ( 0) 0 ( 0) 1 16384 ( 0) 16384 TChi Nonp 4 ( 0) 0 ( 0) 4 16384 ( 0) 4096 TChu Nonp 4 ( 0) 0 ( 0) 4 16384 ( 0) 4096 TChe Nonp 4 ( 0) 0 ( 0) 4 16384 ( 0) 4096 TCht Nonp 2929 ( 0) 2925 ( 0) 4 16384 ( 0) 4096 MmSP Nonp 2 ( 0) 0 ( 0) 2 16384 ( 0) 8192 TChc Nonp 4 ( 0) 0 ( 0) 4 16384 ( 0) 4096 CcBc Nonp 3245000 ( 74) 3244881 ( 28) 119 16184 ( 6256) 136 Nb07 Nonp 73 ( 0) 22 ( 0) 51 15504 ( 0) 304 Wait Nonp 3942538 ( 65) 3942484 ( 65) 54 15368 ( -48) 284 Gsem Nonp 1137 ( 0) 947 ( 0) 190 15136 ( 0) 79 SmMm Nonp 6305 ( 0) 6284 ( 0) 21 14776 ( 0) 703 Nbtl Nonp 13271 ( 0) 13169 ( 0) 102 14688 ( 0) 144 ObjT Nonp 31 ( 0) 0 ( 0) 31 14384 ( 0) 464 NDw1 Nonp 16 ( 0) 6 ( 0) 10 14328 ( 0) 1432 NBSf Nonp 69 ( 0) 52 ( 0) 17 13176 ( 0) 775 PciB Nonp 62 ( 0) 46 ( 0) 16 12776 ( 0) 798 PSC3 Nonp 294 ( 0) 258 ( 0) 36 11848 ( 0) 329 MmLd Nonp 136 ( 0) 33 ( 0) 103 11760 ( 0) 114 IoCo Nonp 1166 ( 0) 1003 ( 0) 163 11736 ( 0) 72 NpFc Nonp 4293 ( 0) 4159 ( 2) 134 10720 ( -160) 80 WmiR Nonp 576 ( 0) 413 ( 0) 163 10432 ( 0) 64 IdeP Nonp 19785 ( 0) 19750 ( 0) 35 10112 ( 0) 288 RxIr Nonp 199440 ( 0) 199429 ( 0) 11 9944 ( 0) 904 NaTa Nonp 37 ( 0) 0 ( 0) 37 9768 ( 0) 264 SdHt Nonp 37 ( 0) 0 ( 0) 37 9768 ( 0) 264 UlRP Nonp 67865 ( 0) 67849 ( 0) 16 9352 ( 0) 584 RxMs Nonp 1200 ( 0) 1198 ( 0) 2 9288 ( 0) 4644 rx Nonp 3 ( 0) 0 ( 0) 3 9024 ( 0) 3008 CcPc Nonp 480765 ( 4) 480677 ( 4) 88 8448 ( 0) 96 ScFt Nonp 8 ( 0) 1 ( 0) 7 8432 ( 0) 1204 FMfr Nonp 1 ( 0) 0 ( 0) 1 8192 ( 0) 8192 Usfd Nonp 1 ( 0) 0 ( 0) 1 8192 ( 0) 8192 AfdL Nonp 798692 ( 34) 798489 ( 29) 203 8120 ( 200) 40 AfdF Nonp 11799 ( 0) 11787 ( 0) 12 7968 ( 0) 664 NbL2 Nonp 4033 ( 2) 3990 ( 1) 43 7912 ( 184) 184 NDpf Nonp 41 ( 0) 15 ( 0) 26 7904 ( 0) 304 SePa Nonp &nAnonymous
May 15, 2008
DIALOGsoftware - a large number for FILE usually indicates that some user-mode process has a large number of handles. Find it and fix it. Likewise for TCPc - large number of sockets being "leaked" by something running on your system. You will have these problems until you fix the software that is leaking the resources. //DavidAnonymous
May 19, 2008
Thanks for this help. I'll try to find this problems and will inform you if i've found the soft that is causing me these troubles.Anonymous
August 25, 2008
The comment has been removedAnonymous
October 16, 2008
Thanks to everyone for their input and observations on these issues. I have been fighting stability issues with Websense 6.3.2 (a web filtering system) in a Windows 2003 Server environment for months. We would end up with a system that could not make ANY kind of network connections due to the exhausted Nonpaged pool. After getting a pointer toward 'tdLL' as a memory leak source from Websense Support, I found this article and the collective experiences and ideas. It appears that SAV9ce was a significant part of our issues, and after upgrading to Symantec Endpoint Protection 11 (11.0.3001.2224 specifically) our servers appear to be much better behaved. Huge thanks to David and all of the thread contributors!Anonymous
October 17, 2008
Tag Type Allocs Frees Diff Bytes Per Alloc File Nonp 920718825 920584429 134396 20523872 152 Even Nonp 9561711 9440980 120731 5798032 48 TCPc Nonp 2604286 2494258 110028 5281344 48 Ntfr Nonp 843921 809887 34034 2179144 64 Mdl Nonp 4991145 4958701 32444 4154432 128 NtFs Nonp 477786384 477765570 20814 835056 40 Ntfn Nonp 876015 855203 20812 834552 40 MmCa Nonp 1315376 1298180 17196 1916240 111 how to find where is mem leaks from first line File Nonp 920718825 920584429 134396 20523872 152 which driver is causing issue ? VishAnonymous
October 21, 2008
Vish - I suggest reading the collective knowledge from earlier in this thread for information on how to diagnose the "File" tag. //DavidAnonymous
February 23, 2009
David - You're my hero! IIS on one of our Exchange servers was refusing connections. Turned out to be Exchange Public Folder replication NPP memory leak. Your How-To worked like a charm in diagnosing our root cause. Thanks!Anonymous
April 19, 2009
http://blogs.technet.com/askperf/archive/2007/03/07/memory-management-understanding-pool-resources.aspx according to the url above, the size of non-paged pool depends on the amount of physical memory present in the system.Anonymous
June 11, 2009
The comment has been removedAnonymous
June 17, 2009
We are facing same problem from last 6 months, i have one doubt that after we have installed addon of url-rewrite this thing is happening, Let me know if anybody else has used that and if that could be cause of problem.Anonymous
July 22, 2009
Thanks David! My web server was using 107mb of 127mb due to SavE consuming 42mb. As soon as i uinstalled symantec and restarted IIS the problem was cleared. It has the 3gb switch and has for over a year now, so I'm not sure what finally triggered it after it being fine for so long.Anonymous
September 12, 2009
Dave, i got MPIO in my poolmon output. MPIO is MS iSCSI initiator, boucing the service didn't help resolve the problem. still not able to see the web page..please advise MPIO Nonp 6088888 ( 12) 31 ( 0) 6088857 170516336 ( 336) 28 [mpio] MmCm Nonp 2209 ( 0) 2045 ( 0) 164 9061648 ( 0) 55253 Unknown Driver AfdB Nonp 5460432 ( 43) 5441960 ( 43) 18472 8881344 ( 0) 480 [afd] tmte Nonp 1949 ( 0) 0 ( 0) 1949 7983104 ( 0) 4096 [tmtdi] TCPB Nonp 459974 ( 0) 412860 ( 0) 47114 4966880 ( 0) 105 Unknown Driver File Nonp 93043247 (2752) 93012719 (2752) 30528 4643120 ( 0) 152 Unknown Driver AfdC Nonp 222392 ( 0) 195975 ( 0) 26417 4226720 ( 0) 160 [afd] Wdm Nonp 2 ( 0) 0 ( 0) 2 3420160 ( 0) 1710080 [ati2mtag][hid TCPC Nonp 73028 ( 0) 46348 ( 0) 26680 2246032 ( 0) 84 [tcpip] TCPt Nonp 3357329 ( 0) 3357294 ( 0) 35 1458688 ( 0) 41676 [tcpip] Mm Nonp 139 ( 0) 114 ( 0) 25 1369536 ( 0) 54781 Unknown Driver TCPc Nonp 526016 ( 1) 499134 ( 1) 26882 1290336 ( 0) 48 [tcpip] LSwi Nonp 2 ( 0) 1 ( 0) 1 1290240 ( 0) 1290240 Unknown DriverAnonymous
May 02, 2010
Now I have this problem. In my case, I found that server restart is not that necessary. Recycling (manually) the application pool, and then restarting the WWW service should work.Anonymous
September 29, 2010
Excellent post Dr. Wang I had that problem since we took over the support for this W2003 server. I did set a maintenance reboot every week and solved the problem. But now it is IIS6.0 that behaves and refuses connections. So my homework brought me to these NPP results : Memory: 3667824K Avail: 732136K PageFlts: 793 InRam Krnl: 2980K P:48752K Commit:4620184K Limit:7194488K Peak:4885544K Pool N:124028K P:50728K System pool information Tag Type Allocs Frees Diff Bytes Per Alloc File Nonp 31021512 ( 169) 30681043 ( 160) 340469 51754600 ( 1368) 152 Even Nonp 20775711 ( 162) 20349506 ( 155) 426205 20464880 ( 336) 48 MmCm Nonp 276399 ( 0) 275249 ( 0) 1150 16413864 ( 0) 14272 R100 Nonp 41 ( 0) 2 ( 0) 39 9683040 ( 0) 248283 Obtb Paged 5021 ( 0) 3294 ( 0) 1727 6716368 ( 0) 3889 R100 Paged 47 ( 0) 2 ( 0) 45 5461800 ( 0) 121373 I found that both File, Even and R100 were linked to ATI related *.sys files. (when running "findstr /m /l File *.sys" under c:windowssystem32drivers ) Then checked the driver used by this ATI Video card driver, it was dated from 2006, and upgraded this ATI ES1000 driver and....Anonymous
November 08, 2010
support.microsoft.com/.../934878 Here is the solutionAnonymous
April 12, 2012
Further research turned up the answer, to the question in my previous comment, here on Mark Russinovich's blog: blogs.technet.com/.../3283844.aspx FYI, (] B [)
