How do you measure Enterprise Architecture?

One thing that I've come to truly appreciate: the balanced scorecard.  Don't get me wrong: I've been using scorecards and dashboards for over a decade.  I helped to build one at American Express.  But I have come to see, from an executive level, why they are so freakin' useful... you can use them to hold people accountable to measurable strategic improvement.

With a scorecard, it is possible to reduce "passion-based decision making" from the organization, without requiring every decision to be based on Return-on-investment.  (I like ROI, but only as a single measure within a balanced scorecard, not as the entire scorecard mechanism ;-).  If everyone understands the mechanisms by which "organizational health" are measured, then it is OK to improve one measure at the expense of another if the final outcome moves toward "health." 

In that vein, I'm looking at the measures that Enterprise Architecture should use to demonstrate alignment with critical IT strategies and business goals.  We have to make sure that our work delivers value, and demonstrate that value, as part of our own scorecard. 

The Corporate Executive Board, an excellent organization that brings together peers from across industry, put together a presentation on the various measures of Enterprise Architecture used by their member companies.  I won't go into details, but it appears that the measures break down into four general areas:

• EA environment and activities -- this is what I call "Proof of Life" metrics.  Useful process metrics, and you can put ranges on them to push general activity.  These kinds of measures include "number of to-be architectures defined," and "number of business processes mapped."  Unfortunately, if a metric is not properly aligned, these metrics can end up being little more than "looking busy."  These metrics prove you are working, but not that the work is having a positive impact.
   
• EA compliance and adoption -- these are the "Proof of Effect" metrics.  This is a lot closer to proving the case that EA is not only present, and busy, but having an effect.  These include measures like "% of applications used by more than one business" and "% of projects compliant with EA standards" and "% of transactions that adhere to master data standards."  Assumably, these are good performance indicators that can be rationally tied to business value.  Having this measure is important.  Having that connection to business value is also important.  Note that the CEB study did not include two of the key measures that Microsoft IT finds important:
  • % of Business Stakeholders that view IT as a trusted advisor and strategic partner, and
  • % of Strategic Project Milestones reached on time 
     
• Spending and Savings -- these are the "Cost Cutting" metrics.  These are directly valuable to the business, as a single dollar of cost saved can go straight to the bottom line.  This group of measures includes things like "savings from a reduction in interfaces," and "savings from standardized purchase agreements."  You often need the "Proof of Effect" metrics to back up this group, to show that there is a correlation.  Otherwise, you can leave open the possibility of having a really large impact, for which another group is given credit.  For those of you involved in getting funding for EA, you'll recognize how perilous that road can be.
   
• Revenue and Profit -- these are the "Value Stream" metrics.  These metrics are valuable to the company's stockholders in the most visible of ways.  Metrics like these can include "revenue from new IT-enabled business capabilities" or "opportunity benefits of agility: revenue during time-to-market savings."  Unfortunately, it can be a long road between "govern the standards or an IT project" and "increase revenue."  At this level, EA can be part of a contribution to IT alignment agility and quality, which can be part of a contribution to Business agility and performance, which contributes to Business profitability.  On the other hand, I think that these numbers are not the better measure of EA performance since the contribution can vary wildly from one project to the next, or even one quarter to the next, due to conditions that are completely outside of the control of EA (or even IT).  In many cases, these measures are the "cinnamon air freshener" of the CIO's office.  They smell nice, but vanish quickly, leaving behind no evidence that they were ever there.

Personally, I found this study useful on so many fronts.  It gave me context, ideas, and key questions to answer.  But now I'd like to ask you, the practitioner... what do you think?

If it were up to you to create a measure of Enterprise Architecture, what metrics would you collect?  What metrics would you ignore? 

Please share...

Comments

• Anonymous
  March 07, 2009
  These are excellent points around the value of a balance scorecard, rather than an ROI metric alone.  How do you see extending these type of metrics with a balanced score card to Government based SOA type projects such as Medicaid and Medicare?

• Anonymous
  March 08, 2009
  The comment has been removed

• Anonymous
  March 08, 2009
  The comment has been removed

• Anonymous
  March 08, 2009
  Hi Nick, The balanced scorecard is definitely an approach of measuring something and it seems you have covered to measure "everything" in the scope of EA. I understand that the objective of measuring EA is intended to cover everything but wouldn't it be better focus on measuring the EA to align it with business benefit? Is there a way to answer the following questions?

1. Can I walk into an organization and use the model to say that it is a maturity (EA) level 3.5 in a scale of 1-5? (And this is what you need to move up to a level 4)... Or your Business and IT are aligned at 0.7 in a scale of 0-1 and this is your roadmap for a better alignment.
2. Is there a way to link an executive dashboard with EA-nugget plug-ins to show the real-time benefit of the traceable monetized transactions or business processes? Which in other words define and measure the business benefits?
3. Can we link the maturity of measuring and governance enforcement with the EA maturity or Business-IT alignment?
4. How do we model the governance (or define a scorecard) to achieve this? These are some questions which are answered in many ways. Is there a way define a reference model for this? I believe that is what you are interpreting here for "Measuring EA", and it would be great to have your thoughts on my questions. Regards, Anirban

• Anonymous
  March 09, 2009
  The comment has been removed

• Anonymous
  March 15, 2009
  Just a quick note. I ran across this interesting post from Leo de Sousa, an Enterprise Architect in the

• Anonymous
  March 17, 2009
  The comment has been removed