The Performance War -- Win it 5% at a time

If it feels like getting good performance out of your application/library/service/whatever is more like "trench warfare" than it is like "shock and awe" then you're probably doing something right.

The trouble with performance work is that the easy work gets all the press.  Well, sort of.  Let me explain. 

Suppose you're an elite performance engineer. Sometimes a big ugly nasty problem gets dropped in your lap. When that happens you roll up your sleeves, apply some good techniques, and maybe you net a nice fat win for your clients.  Everyone cheers and they all rush to the store buy a superhero cape just like the one you wear.  Children ask for your autograph.  It's good times for everyone. 

OK well, maybe I'm getting a little carried away.

The thing is when something like the above happens probably you shouldn't be celebrating at all.  Giant performance wins that could not be achieved without the help of a superhero are usually a sign that something has gone terribly wrong in the design process.  Perhaps the team in question left their performance work to a point that was too late in the cycle.  Perhaps they had some basic flaws that had to be remedied, flaws that never should have crept into the codebase in the first place.  But the fact that some "superhero" was able to come along and significantly fix them points more towards mistakes in the original code than it does to any greatness on the part of the hero.

Still, sometimes that's the gig and so you do it.

Now the "real" performance work, the stuff you should be proud of because it's hard, is much less glamorous.  Basically it happens by carefully understanding your whole process, avoiding any big problems (so that they never require a superhero to come along and fix), and steadily working on the most important areas slowly but surely. 

In a mature product with a healthy process you're much more likely to see a 50% gain come in the form of many 5% gains compounding to get to your goal via sustained effort and quality control.  Those wins are largely unsung but they are the hard ones.  They are the wins that give you headroom for your new features and convert your older features from sluggish to snappy.  Every one of them is hard work.

The tragedy is that more care and effort often goes into any one of those fixes than one superhero action, but the capes get all the good press.

Huge instant performance wins are more often a sign of problems than they are of greatness.

Comments

• Anonymous
  October 17, 2005
  great post, I couldn't agree more
• Anonymous
  October 17, 2005
  Sure. But managements simply can't understand.
  
  They want good improvement figures. If you've persuding your boss to give you time for profiling, they'll surly be much happier to see a 50% improvement than a 5% one.
• Anonymous
  October 21, 2005
  That is a great post! Very interesting! I hope I will some day be able to perform both the "Superhero" type And the "real" performance work, as appropriate/necessary. ;)
  
  Thanks.
• Anonymous
  October 24, 2005
  So to make managers happy, you create a bad desing and then later you fix it. Maybe you'll get even 90% performance improvement. :-) The truth is that educating managers is also part of the process. I think they can understand (it's difficult, I know), maybe we are using the wrong approach. There are some obstinate individuals out there I must admit. How about this, let's try to make them think is their idea.... keep your spirits up...
• Anonymous
  October 27, 2005
  The comment has been removed
• Anonymous
  October 31, 2005
  The comment has been removed
• Anonymous
  November 02, 2005
  Of course if that's 5% bursts across multiple changes and there's a 3 year pause between releases, 50% is possible.
  
  I mean look at some of the XML bragging in .Net 2.0 where somebody here on MSDN blogging showed improvements as large as 400%.
  
  And just recently I came across a chart showing many boosts in the .Net Compact Framework v 2.0, one improvement was a whopping 800%.
  http://www.zintel.net/Blog/NETCFPerfProgress.jpg
  
• Anonymous
  November 02, 2005
  Here's one site showing the massive XML speed improvements with XML in .Net 2.0
  http://msdn.microsoft.com/msdnmag/issues/04/02/XMLFiles/default.aspx
  
  Of course looking back at old code, and having more optimized code, it's easy to see myself calling that old code "a problem".
• Anonymous
  April 18, 2006
  A while ago I wrote about how you often win the performance war 5% at a time.  The theme of that...
• Anonymous
  April 20, 2006
  I remember Raymond Chen gave a talk at the last PDC and had this fun quote:
  
  "One of the questions...
• Anonymous
  April 24, 2006
  
  
  Last
  week I described an optimization that helps Paint.NET's startup performance by
  avoiding our...
• Anonymous
  January 23, 2007
  The thing about performance work is that it's very easy to be fooled into looking into the wrong areas.