A Fresh Look at ROI
By Jay Cross Training return-on-investment meets the information age.
Ask me, I'll tell you: Return-on-investment isn't what it used to be.
This traditional financial measure, developed by DuPont and once credited with making General Motors manageable, hasn't kept pace with the times. The R is no longer the famous bottom line and the I is more likely a subscription fee than a one-time payment.
Until recently, most training decisions were incremental. Training sponsors had most of the infrastructure required: an empty room, staff, flipcharts, markers, perhaps some personal computers. Business unit managers could evaluate the cost-effectiveness of one-shot training courses by assessing cost and effect within their own business units. E-learning changes this, though.
E-learning is a continuous process, not a one-shot deal. It is most often an enterprisewide initiative, beyond the bounds of any individual business unit. And investing in e-learning is often a strategic imperative--the entry ticket to an e-business environment.
Different strokes for different folks
Where you stand on ROI depends on where you sit. Different levels of management make different sorts of decisions, so it's appropriate that they use different measures of ROI.
| Function | Goal | Measurement | Scope |
| Training manager | Close skills gap | Individual performance | Business unit, specific training |
| Business unit manager | Achieve business goal | Project goals, business metrics |
| Corporate staff | Choose the best alternative | Financial metrics, business case | Enterprise, e-learning infrastructure |
| Executive management | Gain competitive advantage, transformation | Business case, shareholder value |
All these goals and measures are valid--in fact, they complement one another. The more each managerial group understands the others, the better the odds they'll make sound decisions.
The entire domain of business decision making is at a crossroads; the old yardsticks no longer apply. Why measure incremental improvements when you're seeking the Holy Grail? Rational decision makers look beyond an ROI that reduces everything to the lowest common denominator. Traditional financial analysis works in stable times but falls apart when things go off-scale.
Bearing that in mind, let's examine ROI from four different perspectives: training manager, business unit manager, corporate staff, and executive manager.
Caveat: Maybe your training manager is a chief learning officer who has the ear of top management, or a wave of re-engineering swept your corporate staff out the door. When these mismatches pop up, substitute the person in your organization who has these interests, positions, and ways of doing things.
The training manager perspective
Don't step in the ROI. Five or six years ago, I witnessed a multimedia course demonstration designed to help bank officers spruce up their table manners. When a banker clicked on sushi, a voice intoned "sooo-she." Then a Japanese flag appeared, followed by words explaining that sushi is a popular Japanese dish made of fish and rice. I chuckled and thought that's going to be one surprised banker when the raw fish arrives.
The sushi story came to mind last fall during a series of breakout sessions at the TechLearn and Online Learning conferences. But this time, the topic was ROI.
Consultants drove home the message relentlessly to dozens of groups: If you want to sell a big project internally, you've got to talk ROI. It's the language senior managers understand, and it's how they separate worthy projects from losers. Being fluent in ROI talk enables you to position a learning project as an investment rather than a cost, they said. It's the secret handshake that gets you into the inner circle of those who control budget dollars.
Well, it's reality-check time. Talking the ROI talk won't enable you to pass yourself off as an astute businessperson. You have the same chance of passing for French with a beret and Berlitz phrase book. A little knowledge can be dangerous. Making a significant business decision entails a wide range of factors and involves intricate tradeoffs. For instance,
- Risks must be weighed against rewards.
- Short-term aims need to be sorted from the long.
- Undertakings must align with strategic initiatives.
- Scarce resources call for shrewd horse-trading.
Unless your training unit sells training for a fee--generating its own revenues--the returns on training investment come from satisfying the needs of business unit managers. Tying training results to business results is more useful than coming up with pseudo-ROI figures.
The only valid training ROI is business ROI.
Mental images trump numbers. International Data Corporation studied the buying behavior of corporate and IT training managers and concluded that, "ROI will no longer be measured in savings or reduced cost of training." Instead, attention will be directed to "measurable changes to business metrics resulting from training investments. Those benefits will only emerge if vendors increase their focus on high-quality instructional design and engaging learning environments."
Often ROI exercises are necessary to show you've done your homework; the numbers are your ticket to the adults' table. A decision maker's final choices, however, are mostly visceral--based on how the likely outcome of each alternative makes him or her feel. Feelings win out because the assumptions used to create the numbers can always be challenged. Projects that evoke the best feelings make the cut.
To sell a project upward, you've got to make the upside come alive--paint a picture, tell a story. And make no mistake about it, upselling is what you need to do.
The business unit manager perspective
Business unit managers own the problems that training solves. They're generally pragmatic, and their overriding interest is getting a job done. Soon. Until you know what they're trying to accomplish, you can't talk with them about potential results.
The business unit manager is usually training's primary sponsor. When you're working with the right client--the decision maker who understands the end goal and controls the environment in which the problem occurs--measuring results can be simple. Start with business problems and work backwards. The most important step in measuring performance is pinning down the business manager's answer to the classic question: What's in it for me?
Don't skip this step. Without it, meaningful tracking is impossible. First gain agreement on the problem and the value of solving it. Then outline how you propose to solve it. Establish a baseline measure of current performance, and clearly indicate how performance will be tracked and reported.
Proof is a figment. Brilliant people assure me that it's impossible to isolate the impact of training. You can never tell whether some concurrent event has contaminated the results and negated their value as scientific evidence of training's impact, they say.
To which I reply, "Baloney!" (Not an exact quote.) Business decisions are made with less-than-perfect information; it comes with the territory. Management is not conducting a science class--it's looking for results.
So the question isn't, "How do we prove beyond a shadow of a doubt that a given training program produced a given result?" Rather, it's "What will our sponsor accept as persuasive evidence that the program produced the result?" Working with strong probabilities, make the case logically, linking learning to business results. Establish a causal link between a particular skill deficiency and a particular business outcome. If the manager buys into this logic and the way you will measure requests after training, that will be the proof you need.
How to track business unit results. The process of tracking learning results starts before any learning takes place. It begins with partnering between the training manager and the business unit manager who owns the problem. So, it's important to agree on the value of solving the problem.
In most cases, you gather information through interviews that focus on the work process, not training. And they should always return to the mother of all business questions: What difference would this make?
A joint examination of the problem will pinpoint the gap between the results the manager wants and the results he or she will actually get. Then, determine what major skill gaps and learning deficiencies might hold people back.
Next, estimate the expected dollar value gained by eliminating the deficiency and make tangible projections from those outcomes. Make sure to get signoff on the expected outcomes, how they'll be measured, and performance criteria. This keeps discussions focused and agreements documented.
Meanwhile, throughout the process, you're helping managers answer questions about why skills matter and what good performance looks like. You're focusing sustained attention on solving problems and adding value, and you're identifying tangible values for each skill to be taught.
As a result, you're forging a partnership with the business unit client based on his or her core concern: performance.
When learning has been completed, assess the results according to measurements set up with the manager. Extrapolate behavior changes into measurable business. There's no room for vagueness--and no backing away from visible quantitative evidence. Further interviewing and a review of business results may be useful.
Finally, present your findings and a simple cost-benefit analysis to the business manager or training sponsor.
The corporate staff perspective
No organization has the resources to do all the good things it might; senior executives must choose where to place the company's bets. A major role of corporate staff is helping execs make sound choices. Thorough analysis reduces the risk of choosing the wrong training alternative. The bigger the project, the more analysis is justified.
For clear analysis, an effective staff distills a complex business alternative into a three- or four-page business case. Solution Matrix's Marty Schmidt develops business cases for corporations, and he puts it this way: "A business case is a tool that supports planning and decision making, including decisions about whether to buy, which vendor to choose, and when to implement. It is generally designed to answer the question: 'What are the likely financial and other business consequences if we take this or that action (or decision)?'"
The Solution Matrix Website shares tips on how to assemble an effective business case. Here are a few words of advice.
- Explain where the data comes from to maintain credibility. Describe any assumptions the reader wouldn't make automatically.
- Break out "hard" benefits from "soft," because some readers will value them differently. The difference? It's easy to assign a dollar figure to a hard benefit (for example, reduce travel expense by $180,000) but difficult for a soft benefit (free up professional time).
- Develop a statement of the project's net cash flow.
- Quantify every benefit and cost possible.
- If an item simply can't be quantified ("Morale will improve," for example), include it in a nonfinancial analysis, and rank it among the financial impacts to show its relative importance.
- Include a sensitivity analysis to show what happens when assumptions change and a risk analysis to show that likelihood.
- Assume that the numbers don't tell the whole story. Make the case that the project boosts sales, improves service, speeds things up, and so forth, then relate those benefits to business objectives.
Because a business case uses financial metrics (ROI, internal rate of return, discounted cash flow, and so forth) and numbers are precise, it's tempting to imbue financial metrics with too much importance. Numbers are another perspective, not another reality.
Beware of bad numbers. Present-day accounting is an anachronism. It worked well when people could go out to the warehouse and count assets. In the information age, it's an inappropriate yardstick because most of the assets drive home every night.
In a nutshell, traditional accounting recognizes physical entities; intangibles are valued at zero. Vast areas of human productivity--ideas, abilities, experience, insight, esprit de corps, motivation--lie outside its vision field. It doesn't recognize that people become more valuable over time.
Your gut may tell you that you'll be repaid in the future for investing in people today, but where training is concerned, the only metric taken into consideration is cost. Accounting has no measuring stick to distinguish a good idea from a bad one. Excellent training hits the books at the same value as bad.
What alternative is there? Weigh subjective factors as well as objective ones. Keep two sets of books, one for investors and regulators, the other to track the intangibles.
Robert S. Kaplan and David P. Norton, co-authors of The Balanced Scorecard, devised a means of evaluation developed to help make up for the insufficiencies of financial accounting. In addition to finances, the balanced scorecard looks at changes in customers, processes, and employees. The method was designed to look backward--to evaluate in hindsight--but there's no reason not to use it to project into the future as a decision-making tool.
According to Kaplan and Norton, managers have traditionally attempted to improve performance by making operating and investment decisions to develop new and better products, to increase sales, and to reduce operating costs. "Over time, however," they write, "it probably occurred to some managers that during difficult times, when sales were decreasing and operating costs were increasing, profits could be earned not just by selling more or producing for less, but by engaging in a variety of nonproductive and typically nonvalue-creating activities."
The executive perspective
Top management is committed to implementing strategic changes to transform the enterprise and increase shareholder value. But the bottom line--profits left over after subtracting costs from revenues--is no longer the be-all and end-all of business results.
Executives typically focus on two things: strategy and outfoxing the competition. They realize that competing successfully requires teams of inspired employees who are mentally equipped to make sound decisions on the fly; able to execute good ideas in a snap; and are proactive when it comes to taking initiatives and bringing innovation. The overall goal: an environment where people learn faster and better than the competition.
Getting there takes more than a lavish investment in training, though. Time is frequently more important than money. According to Klaus Schwab, founder and president of the World Economic Forum: "We are moving from a world in which the big eat the small to a world in which the fast eat the slow."
Let's look at how senior decisions are made. The staff has shopped various projects, gathered the figures, done due diligence on suppliers, run the numbers, assessed the impact of changes in the marketplace, and prepared terse summaries for each scenario. A couple of projects are no-brainers. These are so integral to the organization's mission that giving a go-ahead is a formality.
Projects that enter new territory--e-learning for example--warrant more detailed consideration, though.
My experience has shown that most senior executives have more faith in gut feeling` than in numbers. The numbers are input, but the decision is broader than that. Results from an Information Week survey reveal that "More companies are justifying their e-business ventures not in terms of ROI but in terms of strategic goals. Creating or maintaining a competitive edge was cited most often as the reason for deploying an application."
To some people--me included--the traditional concept of training ROI is obsolete. Astute training managers employ business metrics, not evaluation levels, I believe. Business unit managers value time more than ROI. Major decisions are based on descriptive business cases, not pro forma budgets. Senior executives tend to be more interested in the top line (dramatic growth from new markets and innovation) than the bottom line (the accounting fiction of profits).
The Net has changed everything.
Published: January 2001
