ASP.NET Memory - How much are you caching? + an example of .foreach
I was writing a post on debugging a high memory usage problem caused by storing too much in session scope, but I realized I got in to tangent discussions all the time so I decided to create a separate post on caching first.
Caching is by far the most common cause of high memory problems. It’s an understatement to say that the cache is often overused.
As with everything else nothing comes for free, so the performance you gain by storing items in cache has to be compared to the performance you loose by having to scavenge this memory. Be careful with the design and set expiration times etc. appropriately, and make sure that what you cache will actually be accessed enough times to make it worth while.
Sos.dll has some nice features for examining your cache through the !dumpaspnetcache command; however before we get in to that, let’s look at some commands to determine if you have a cache problem.
Where is the cache?
Each HttpRuntime (running application) has a reference to a System.Web.Caching.CacheMultiple object (the _cache field), where the cached objects for that application is stored. In the particular version I am running (1.1.4322.2300), the _cache field is located at offset 0xc from the address of the HttpRuntime object.
0:023> !do 0x061b47d4
Name: System.Web.HttpRuntime
MethodTable 0x01b29dfc
EEClass 0x01ee8314
Size 116(0x74) bytes
GC Generation: 2
mdToken: 0x02000079 (c:\windows\assembly\gac\system.web\1.0.5000.0__b03f5f7f11d50a3a\system.web.dll)
FieldDesc*: 0x01b29554
MT Field Offset Type Attr Value Name
0x01b29dfc 0x4000683 0x4 CLASS instance 0x00000000 _namedPermissionSet
0x01b29dfc 0x4000684 0x8 CLASS instance 0x061b4950 _fcm
0x01b29dfc 0x4000685 0xc CLASS instance 0x061b4ca0 _cache
0x01b29dfc 0x4000686 0x54 System.Boolean instance 0 _isOnUNCShare
0x01b29dfc 0x4000687 0x10 CLASS instance 0x061b6e18 _profiler
0x01b29dfc 0x4000688 0x14 CLASS instance 0x061b6e34 _timeoutManager
0x01b29dfc 0x4000689 0x18 CLASS instance 0x021b368c _requestQueue
0x01b29dfc 0x400068a 0x55 System.Boolean instance 0 _apartmentThreading
0x01b29dfc 0x400068b 0x56 System.Boolean instance 0 _beforeFirstRequest
0x01b29dfc 0x400068c 0x60 VALUETYPE instance start at 0x061b4834 _firstRequestStartTime
0x01b29dfc 0x400068d 0x57 System.Boolean instance 1 _firstRequestCompleted
0x01b29dfc 0x400068e 0x58 System.Boolean instance 0 _userForcedShutdown
0x01b29dfc 0x400068f 0x59 System.Boolean instance 1 _configInited
0x01b29dfc 0x4000690 0x50 System.Int32 instance 1 _activeRequestCount
0x01b29dfc 0x4000691 0x5a System.Boolean instance 0 _someBatchCompilationStarted
0x01b29dfc 0x4000692 0x5b System.Boolean instance 0 _shutdownInProgress
0x01b29dfc 0x4000693 0x1c CLASS instance 0x00000000 _shutDownStack
0x01b29dfc 0x4000694 0x20 CLASS instance 0x00000000 _shutDownMessage
0x01b29dfc 0x4000695 0x68 VALUETYPE instance start at 0x061b483c _lastShutdownAttemptTime
0x01b29dfc 0x4000696 0x5c System.Boolean instance 1 _enableHeaderChecking
0x01b29dfc 0x4000697 0x24 CLASS instance 0x061b7014 _handlerCompletionCallback
0x01b29dfc 0x4000698 0x28 CLASS instance 0x061b7030 _asyncEndOfSendCallback
0x01b29dfc 0x4000699 0x2c CLASS instance 0x061b704c _appDomainUnloadallback
0x01b29dfc 0x400069a 0x30 CLASS instance 0x00000000 _initializationError
0x01b29dfc 0x400069b 0x34 CLASS instance 0x00000000 _appDomainShutdownTimer
0x01b29dfc 0x400069c 0x38 CLASS instance 0x061d4b24 _codegenDir
0x01b29dfc 0x400069d 0x3c CLASS instance 0x02188f1c _appDomainAppId
0x01b29dfc 0x400069e 0x40 CLASS instance 0x02188f68 _appDomainAppPath
0x01b29dfc 0x400069f 0x44 CLASS instance 0x0218a434 _appDomainAppVPath
…
I’m not going to go in too much in the nitty-gritty but basically this CacheMultiple has a _caches Object[] member variable, which contains some CacheSingle’s and they each have an _entries membervariable which contains the actual CacheEntries, if you want to delve in to them manually. Luckily for us, !dumpaspnetcache automates this as you will see later.
Getting the size of the cache
The reason that I still mention where the cache is stored, is because with this knowledge we can now compose a command that goes through all our HttpRuntimes and dumps out the name of the app and how much it stores in cache.
The pseudo-code for doing this would be:
For each (HttpRuntime in (all the runtimes on the heap))
Print “*******************” //to divide the entries
Dump out the AppDomainID
Print “Cache Size: “
Dump out the size of the cache
End for each
Translated to windbg terms:
To find all the runtimes on the heap, first find the methodtable for System.Web.HttpRuntime and then run !dumpheap –mt .
First run !dumpheap –type HttpRuntime which gives us all objects whose name contains HttpRuntime and their method tables (MT).
0:023> !dumpheap -type HttpRuntime
Using our cache to search the heap.
Address MT Size Gen
0x061d4dbc 0x020c0010 12 -1 System.Web.Configuration.HttpRuntimeConfigurationHandler
0x021b321c 0x0209ff40 52 -1 System.Web.Configuration.HttpRuntimeConfig
0x061b47d4 0x01b29dfc 116 -1 System.Web.HttpRuntime
Statistics:
MT Count TotalSize Class Name
0x020c0010 1 12 System.Web.Configuration.HttpRuntimeConfigurationHandler
0x0209ff40 1 52 System.Web.Configuration.HttpRuntimeConfig
0x01b29dfc 1 116 System.Web.HttpRuntime
Total 3 objects, Total size: 180
And then do !dumpheap –mt 0x01b29dfc –short
0:023> !dumpheap -mt 0x01b29dfc -short
0x61b47d4
The short switch gives us only the address of the objects so we can use it more easily with .foreach. In this case I had only one HttpRuntime because I have only one vdir that I have accessed since the start of the process.
The AppDomainId (IIS name for the app) is stored in the _appDomainAppId member variable for HttpRuntime at offset 0x3c...
0x01b29dfc 0x400069d 0x3c CLASS instance 0x02188f1c _appDomainAppId
...so to dump this out given a HttpRuntime address you would run
0:023> !dumpobj poi(0x61b47d4+0x3c)
String: /LM/w3svc/1/root/MemoryIssues
The poi operator gives us a dword stored at the address we are giving it, i.e. in this case it will give us what is stored at 0x61b47d4+0x3c, where 0x61b47d4 is the address of the HttpRuntime, so it will give us 0x02188f1c, which is the address of _appDomainAppId from above.
The reason we use this technique is so we can automate it to run on all the HttpRuntime objects on the heap, since with .foreach where we will only know the address of each HttpRuntime object.
Much the same way we get the size of the cache by running !objsize on the _cache member variable
0x01b29dfc 0x4000685 0xc CLASS instance 0x061b4ca0 _cache
0:023> !objsize poi(0x61b47d4+0xc)
sizeof(0x61b4ca0) = 1,336,709,120 (0x4fac9000) bytes (System.Web.Caching.CacheMultiple)
Now we have all the components to run a command that will give us the name and the cache size for all HttpRuntime objects.
A short comment on .foreach before we continue...
In its simplest form the .foreach command looks like this:
.foreach ( Variable { InCommands } ) {
OutCommands
}
Where InCommands is the commands that will give you the array/collection to loop through, and the OutCommands is a semicolon separated list of what commands you want to run each iteration.
Our command will look like this:
.foreach (runtime {!dumpheap -mt 0x01b29dfc -short}){.echo ******************;!dumpobj poi(${runtime}+0x3c);.echo cache size:;!objsize poi(${runtime}+0xc) }
We have to enclose runtime (our foreach variable) in ${} so that it doesn’t get interpreted as a symbol.
And the output looks like this:
******************
String: /LM/w3svc/1/root/MemoryIssues
cache size:
sizeof(0x61b4ca0) = 1,336,709,120 (0x4fac9000) bytes (System.Web.Caching.CacheMultiple)
If I would have had more than one application it would have listed all of them one after another, separating them with a line of ***
So here we can see that my one application MemoryIssues, has a cache that contains 1,336,709,120 bytes. WOW!!! that's a lot!!! We definitely seem to have some sort of cache problem.
What is in my cache? Using !dumpaspnetcache…
!dumpaspnetcache –stat gives you a nice overview over what types of objects you are storing in cache. (check the post on !dumpheap –stat to understand TotalSize).
0:023> !dumpaspnetcache -stat
Going to dump the ASP.NET Cache.
MT Count TotalSize Class Name
0x0211cc9c 1 20 System.Web.Security.FileSecurityDescriptorWrapper
0x020c242c 2 56 System.Web.UI.ParserCacheItem
0x0206c66c 5 260 System.Web.Configuration.HttpConfigurationRecord
0x0c2e7014 1 316 System.Web.Mobile.MobileCapabilities
0x79b94638 4 376 System.String
0x0c2eaeb4 151 7,248 System.Web.SessionState.InProcSessionState
Total 164 objects, Total size: 8,276
!dumpaspnetcache –s gives you a summary of your objects with priority (removable or NotRemovable etc.) along with create time, expires and last updated which can be useful to see if you are caching something for too long.
The ASP.NET 1.1 cache uses an LRU (Least Recently Used) model for removing objects from cache if memory is needed, for the objects that are not marked NotRemovable.
In this example we can see for example some InProcSessionState items (sessions) that started around 15:31:36 and expire on 15:51:36 (after 20 min) and during that time they can not be removed (NotRemovable) by someone trying to clean up the cache to lower memory usage.
0:023> !dumpaspnetcache -s
Going to dump the ASP.NET Cache.
Address MT Priority Create Time Expires Last Updated Key Class Name
0x061d2658 0x0206c66c Normal 01/25/2006 15:31:35 12/31/9999 23:59:59 01/25/2006 15:31:35 System.Web.HttpConfig: System.Web.Configuration.HttpConfigurationRecord
0x021b1acc 0x0206c66c Normal 01/25/2006 15:31:35 12/31/9999 23:59:59 01/25/2006 15:31:35 System.Web.HttpConfig:/MEMORYISSUES System.Web.Configuration.HttpConfigurationRecord
0x061b3f7c 0x79b94638 Normal 01/25/2006 15:31:35 01/25/2006 15:41:35 01/25/2006 15:31:35 ISAPIWorkerRequest.MapPath:/MemoryIssues/StoringStuffInSessions.aspx System.String
0x021b503c 0x0206c66c Normal 01/25/2006 15:31:35 12/31/9999 23:59:59 01/25/2006 15:31:36 System.Web.HttpConfig:/MEMORYISSUES/STORINGSTUFFINSESSIONS.ASPX System.Web.Configuration.HttpConfigurationRecord
0x021b80b4 0x0211cc9c Normal 01/25/2006 15:31:36 12/31/9999 23:59:59 01/25/2006 15:31:36 System.Web.Security.FileSecurityDescriptorWrapper:c:\inetpub\wwwroot\MemoryIssues\StoringStuffInSessions.aspx System.Web.Security.FileSecurityDescriptorWrapper
0x021bc74c 0x020c242c NotRemovable 01/25/2006 15:31:36 12/31/9999 23:59:59 01/25/2006 15:31:36 System.Web.UI.TemplateParser:/MemoryIssues/StoringStuffInSessions.aspx System.Web.UI.ParserCacheItem
0x022028fc 0x0c2eaeb4 NotRemovable 01/25/2006 15:31:36 01/25/2006 15:51:36 01/25/2006 15:31:37 System.Web.SessionState.SessionStateItem:x4ud0tb53l2wfo554aw0yny4 System.Web.SessionState.InProcSessionState
0x02202a10 0x0c2eaeb4 NotRemovable 01/25/2006 15:31:36 01/25/2006 15:51:36 01/25/2006 15:31:37 System.Web.SessionState.SessionStateItem:iw0thd55js0d44qjm2cwao45 System.Web.SessionState.InProcSessionState
0x02202cfc 0x0c2eaeb4 NotRemovable 01/25/2006 15:31:36 01/25/2006 15:51:36 01/25/2006 15:31:37 System.Web.SessionState.SessionStateItem:2r1dtbzh00tanc451pejqqrl System.Web.SessionState.InProcSessionState
…
And finally, if you want to drill down to what cache objects were largest you can use the .foreach again
0:023> .foreach (cachedObject {!dumpaspnetcache -short}) {!objsize ${cachedObject}}
sizeof(0x626b1b8) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6268284) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6265318) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x62623ac) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x625f440) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x625c4d4) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6259568) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x62565fc) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6253690) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6250724) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x624d7b8) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x624a84c) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x62485e8) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6248334) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6248080) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x6247dcc) = 12,000,772 (0xb71e04) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x62479a8) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
sizeof(0x62476bc) = 8,000,812 (0x7a152c) bytes (System.Web.SessionState.InProcSessionState)
…
So in this particular case, most of our cache is used up by large sessions (~12 and 8 MB each), given this knowledge we would of course drill down into the sessions and find out what is in them, but I’ll leave that for my next post.
Phew!:) At last I have the prerequisites for the post I’m writing on sessions
Until then…
Comments
Anonymous
January 26, 2006
Wouldn't it be nice if we could hook into a static readonly Cache.TotalSize, Application.TotalSize and Session.TotalSize ??
KarlAnonymous
January 26, 2006
This post doesn't display well at 1024 width because of the fixed width no wrap fonts in your blue boxes, perhaps you should lower the font size 1 or 2 notches to be more 1024 width friendly.
PS: I'm a page layout coder, 1024x1280.Anonymous
January 26, 2006
Hi Travis,
I'd be happy to change it, because i find it a bit annoying too that some boxes stretch way out on the right hand side... but unfortunately I know practically nothing about CSS:(
my css override for the blue sections look like this
pre.debug
{
font-family: Courier;
font-size: 6pt;
background-color: #99CCFF;
text-align:left;
}
but the 6pt seems to have no effect. If you have any hints on what i'm doing wrong, im all ears:) always interested in learning new stuff:)Anonymous
February 14, 2006
I see you've gone for font size="2" within the pre tag... If you want to use the font tag you could also try this in your CSS
pre.debug, pre.debug font
{
font-family: Courier;
font-size: 6pt;
background-color: #99CCFF;
text-align:left;
}
or was the font tag and size just a workaround anyway?
All that selector does is say you want to apply the style to elements matching either pre.debug or pre.debug font.Anonymous
February 14, 2006
Hi Jonno,
The only difference i see between your tag and mine above is the additional ,pre.debug font
It doesnt seem like it makes a difference when i change it:(
Thanks
TessAnonymous
February 16, 2006
The comment has been removedAnonymous
February 16, 2006
Hi Howard,
In this particular post, the difference between our outputs with !dumpheap -type is that i am running this on a single processor machine while you are on a multiproc machine.
On single processor machines we run with the workstation version of the GC and only use one heap. On a multi-proc machine you would run the server version and have one heap per processor or if you run a hyperthreaded multiproc you would have 2 heaps per processor.
!dumpheap -type shows specifics for each heap. !dumpheap (without -type) doesnt specify per heap.
If you notice in some of my other posts where i am running !eeheap -gc (showing multiple heaps), i am running those samples on a multiproc machine.
!eeversion will show you which runtime version you are running and how many heaps it is using.
Hope this clarifies things...
TessAnonymous
February 19, 2006
The comment has been removedAnonymous
February 19, 2006
Thank you very much Jonno, really appreciate the help...
That seemed to do the trick:) I've now been able to change it to 8pt which is about as low as i can go and still have it readable.
You learn something new every day...Anonymous
March 21, 2006
I'm a little confused by the demo in your post. You show us with the .foreach command that there is a cache object for each runtime but then when you demo the !dumpaspnetcache -stat it only list one group of objects. Are those all objects in all cache objects or only the objects in the first cache?Anonymous
March 21, 2006
Hi Steve,
In this particular sample there was only one application so only one cache, if there would have been more than one cache all objects from all caches would have been listed with the !dumpaspnetcache -stat commandAnonymous
March 07, 2007
This is part four in the ASP.Net tips series . In particular, this is a follow-up to, " ASP.Net QuickAnonymous
March 08, 2007
In the .Net from 2.0 there is no !dumpaspnetcache command. Do you know if its called something different now, or if there is a way to do something similar?Anonymous
March 08, 2007
Hi Jason, There isn't a !dumpaspnetcache command yet in 2.0 so the only way you can do it is through manually dumping out hte contents of System.Web.Caching.Cache but a few common tasks so you dont have to run though the whole ordeal of dumping it out, would be to do the following
- Find the System.Web.Caching.Cache MT in !dumpheap -stat
- !dumpheap -mt on that MT
- !objsize on the objects so that you can see how much you are saving in cache For session state, find the MT for the InProcSessionState objects and run .foreach (Session {!dumpheap -mt <MT> -short}){!objsize ${Session}} Which will give you the size of the individual Sessions and then from there you can drill down into the particular session object that is causing you grief. Hope this helps, Tess
Anonymous
August 13, 2007
The comment has been removedAnonymous
March 13, 2008
How I lost my WinDbg virginityAnonymous
June 04, 2008
The comment has been removedAnonymous
June 04, 2008
Not sure I follow, What is at address 22d2f00 and what is at offset 4c? I.e. what would you get if you did !do 22d2f00Anonymous
October 18, 2013
Hi Tess! i know this is an old post but I was wondering: is the cache stored only in heapor is stored in unmnaged memory? thanks i love your blog!