Next Practices

I read an excellent article today by Paul Kearns called “All you need to know about training evaluation in about 700 words.” I’ll respond with about 800 words. The article resonated with me because I have spent most of my adult life in some role related to training or education. I’ve been in a million conversations about evaluating training and have been subjected – as learner and trainer – to various and sundry attempts to evaluate training.

Kearns’ point in the article is that the Kirkpatrick levels of evaluation that learning organizations have discussed for decades have always been and will always be inadequate, even if you try to amend them by throwing in the more recently added fifth level for ROI. (Just in case you’re not familiar with Kirkpatrick’s levels, Kearns summarizes the original four levels as 1) reaction to training, 2) testing learning, 3) applying learning in the workplace and 4) business impact.)

Why the inadequacy of Kirkpatrick? Because the point of any training in the workplace is to improve business performance in some way, and unless you have first adequately measured and recorded a baseline of that performance before training, then you have no basis for rightly evaluating the effectiveness of the training afterward. I would also add that you must be able to separate the impact of, for example, learning that happens as a result of attending a class or completing an online module or asking colleagues via a social network or reading on one’s own from learning that occurs in other contexts, or else your quantitative calculations will be tainted with unverifiable assumptions that your training is the reason for the difference.

What about the smile sheets typically given at the end of a training event (level 1)? Forget them. Don’t use them. They don’t matter because management didn’t send you to training to feel good or to like the instructor or to have a comfortable classroom environment or to enjoy the refreshments.

How about the pre-tests/post-tests that prove someone can parrot back words and phrases or perform some action the way you want them to following training (level 2)? Forget them. Don’t use them. They don’t matter because they don’t provide any evidence that such knowledge made a difference to the business.

What about efforts to determine whether the learner applied learning in the workplace (level 3)? We’re getting warmer, but there is no guarantee that doing what one was told to do will have a positive impact on the business. Bad advice and bad training happen every day. We have all experienced it.

It is not until you talk about business impact that you even start talking the language of upper management. So unless you are evaluating your training in terms of business impact (ROI, workplace performance and productivity), your measurements are meaningless to the business. And unless you had that baseline performance measured before your training, the measurements afterward are still meaningless.

It is stunning that most training departments don’t even attempt to measure true business impact and that businesses let them continue to get away with it. It is amazing to me that even many professionals in training departments will produce a deer-in-the-headlights look when asked to provide even the baseline performance numbers, much less the post-learning stats. Asking learners or their managers after training for their perceptions of impact apart from measurable criteria is inadequate and too subjective.

Don’t assume incorrectly that I’m against the presence of training/learning/development professionals and their efforts in organizations. I was in that type of role for most of my adult career. I generally find the professionals to be kind, giving, concerned individuals who want to help others and have chosen their profession as a way of doing so. But that doesn’t mean they get a pass on proving their worth to the business.

Kearns suggests a simple, 2-level evaluation model: 1) baseline evidence, and 2) business impact measured against the baseline data. That’s it. Forget smile sheets (level 1). Forget Kirkpatrick. Forget “higher” levels vs. “lower” levels of evaluation. Gather baseline evidence and then do things that positively impact subsequent measures of that data. Stop doing things that don’t positively impact that data.

If I was in a position in a business to affect how training departments measure their impact, I would demand that Kearns’ simple approach be followed and that the next person who talked with me about Kirkpatrick be flogged (ok, maybe I couldn’t get away with that last one). For the training department that continually resisted or refused to provide hard, real data proving their impact, I would replace those people or shut down the department and just let people find and use their own preferred sources of performance improvement. I’m sure the training professionals shown the exit would yell and scream, but I’m certain they couldn’t produce the data to challenge my business decision to eliminate them.