Measuring Learning Effectiveness
Organizations globally are investing $400 billion annually in training. Naturally, if you invest so much...Read More
Organizations globally are investing $400 billion annually in training. Naturally, if you invest so much in training, you should show a measurable positive impact. That’s where training evaluations come in. But are all training evaluations the same? And are the benefits of training evaluations worth the effort? We answer these questions and more by providing examples of training evaluations you can apply in your specific situation.
Training evaluation was defined by A.C Hamblin as “any attempt to obtain information on the effects of training performance and to assess the value of training in the light of that information.” That’s a pretty broad definition, albeit true. And such a broad definition doesn’t provide a lot of guidance to create an evaluation of a training course.
So, the idea of training evaluations is simple, conceptually. Your organization has developed a training program and wants to see its success. But the devil is in the details.
While the definition of a training evaluation isn’t complicated, measuring the impact of training isn’t so easy. There are examples of training evaluations that focus on the financial costs of changing behavior. And others that incorporate holistic measures of skill, ability, and knowledge. And in all fairness, isolating the impact of training is no straightforward task. For instance, an employee’s improved performance could result from several factors. Including office culture, financial perks, supervisory feedback, and growth possibilities. As well as the training they have recently taken.
For instance, perhaps you have designed examples of training evaluations for courses on improving employees’ productivity. Or the efficiency of procedures at the workplace. But significantly different training evaluations are necessary for other situations. For example, consider a course on selling instructional products to generate revenue or educating a customer about a new product they have purchased from you. These are all need different examples of training evaluations. So, finding a suitable evaluation method is essential. To this end, we focus on examples of training evaluations for these and other situations. But first, what are the benefits of training evaluations, and are they worth the effort?
While there are many examples of training evaluations, designing the right one isn’t a trivial task. But before we get into specific examples, let’s highlight the benefits. As you’ll see below, the benefits of training evaluations are multidimensional.
One of the main benefits of training evaluations is pinpointing shortcomings and areas for improvement. What was missing? What was irrelevant? Should we restructure the format?
From the content to the planning and tools used to deliver the courses, slight tweaks can improve results. Moreover, with the fast-paced world of technology, ideas can quickly become out-of-date. And so, it needs updating. All examples of training evaluations must provide recommendations on improving or updating your course.
Another of the benefits of training evaluations is that the feedback can be used to increase the engagement of future participants. By including a few simple questions in your assessment, you can gain valuable insights.
Asking open-ended questions allows you to receive more feedback than asking closed-ended questions. Genuine feedback from open-ended questions can be precious. With it, you can find how to improve the training further. And are likely to identify improvements you wouldn’t have thought of, which will lead to enhancing participant engagement.
You can also use some of the positive responses from participants as testimonials to encourage others to take the training. And these outcomes from the training evaluations will build credibility in the course.
One of the main benefits of training evaluations is to find out if the instructional goals of your training have been met or not. Typically, an assessment of knowledge transfer is assessed at different points throughout the course. And the specific approach will vary depending on the goals of the training. For example, learning specific knowledge and tasks will be easier to gauge. But, the success of less tangible outcomes, such as from a leadership course, may need to be observed in a role-playing situation. Or following completion of the training by assessment later. Such as a supervisor observing how a person interacts with their team members.
No matter the type of training course, you should always include one question in any examples of training evaluations you use. “What are you – the learner – going to do with this new training?”
As firms seek growth opportunities, so do employees. They search for professional development and better pay. Investing in training signals that the company is also focused on their growth. And it has demonstrated that organizations providing training and development opportunities increase retention. So, quality training evaluations also improve employee retention by enabling future improvements in the training.
A focus of most organizations is the efficient use of their resources. And if used correctly, one of the benefits of training evaluations can be to enhance that. No one wants to continue offering a course that isn’t meeting expectations. So, find a way to improve it or stop running it. And divert those resources to areas that have a more beneficial outcome.
You can also use training evaluations to analyze the effectiveness between different training programs or methods of delivery. For example, how does an on-demand e-learning course compare to an in-person instructor-led course? How does the achievement of the instructional goals compare? How does the ROI compare? Do participants prefer one delivery method over another?
Another one of the less common benefits of training evaluations is that they can help you receive extra funding. Use the results of your training evaluations to support requests for an increase in your budget to expand the program. Or, if you are a non-profit organization, apply for a grant to scale or expand the training you are providing.
If you use an instructor in part of your training, a benefit of training evaluations is that you can also provide feedback for them. Are there any areas for future improvement for the instructor or facilitator?
Despite the benefits of training evaluations, there are several common reasons that they fail. These include the following.
For training evaluations to be successful requires at least some feedback from the participants. If the participants aren’t willing to spend the time to provide feedback, you are missing out on a lot of valuable information.
Often people are more willing to provide feedback if the time required to complete training evaluations isn’t extensive. Also, if participants understand there is a genuine interest in receiving their feedback and know why giving it is essential. It’s much more likely they will take the time to complete it.
If evaluations are complex, people will focus on getting through them rather than on the quality of the feedback they provide. And you’ll lose many of the benefits of training evaluations.
If you ask the wrong type of questions, you could miss out on many valuable insights. It’s easy to ask participants to complete a few multiple-choice or rate on a scale type questions. And it’s quick for them to respond. But is it getting at the information you need to improve the course and measure its effectiveness? Often the most valuable information is from an open-ended question. So in creating training evaluations, you have to be precise in the questions you ask and balance what you ask with the ease of completing it.
You could have the best training evaluations ever designed. But unless you act on the information they provide, it won’t help. You must use this valuable information to make decisions about the training.
Below we discuss five very different examples of training evaluations. Each of these provides insights on how to develop different training evaluations. Our first example is training employees in a retail setting on new products.
You have probably heard of the Kirkpatrick method of training evaluations. While well known, this is only one method to evaluate training effectiveness. This method is suitable to use where there are very transactional aspects to the training. Such as in a training session on new products in a retail store selling garments in a clothing store.
It consists of 4 levels of training evaluation described below.
Level 1: Reaction
The degree to which participants find the training favorable, engaging, and relevant to their jobs
Level 2: Learning
The degree to which participants acquire the intended knowledge, skills, attitude, confidence, and commitment based on their participation in the training
Level 3: Behaviour
The degree to which participants apply what they learned during training when they are back on the job
Level 4: Results
The degree to which targeted outcomes occur as a result of the training
Let’s assume that the training consists of a 1 to 2-hour session to explain a line of new clothing products to store sales reps. In particular, you want the sales reps to understand the new clothing line. And the advantages of this line over other products to increase sales.
The first part of your training evaluations could be a few questions the participant rates from 1-10 to assess their reaction to the training. Was the session informative? Are they excited by the new product line etc.?
The next part of the training evaluation could be a short quiz testing their knowledge of the products. This could be using multiple-choice questions to test knowledge about the features of the products. Or advantages of them over similar products.
The third part determines if they can apply the knowledge they have learned when they are back working. One way to do this is to have their supervisor observe their interactions with customers. The added benefit of this is that the supervisor can also provide coaching and feedback.
The final part of the evaluation of training process assesses how well the training helps increase sales of the new products. To be effective, you need to have baseline data about product sales before the training. Or, if this is an entirely new product, you could have a few control groups. These are groups of sales reps who aren’t given the training, and you compare their sales with those of reps who have taken the training. Based upon these results, you can assess how well the training contributed to the sales goals.
A better model for training evaluations in less transactional situations is Will Thalheimer’s LTEM. This model is described below.
Let’s apply this model to an employee diversity training session. These examples of training evaluations could be used in similar situations. In this instance, the main goal of this program is to create a positive work environment by helping employees recognize and be tolerant of differences among co-workers. Training evaluations of this type could be constructed as follows.
Tier 1 – Attendance. This is simply a measure of how many people (or percentage) attended the training session.
Tier 2 – Activity. For example, how many people started the course, and how much of it did they complete?
Tier 3 – Learner Perceptions. The part of the training evaluations could be in the form of a survey about the course. This could include feedback on a ranking scale about their perceptions. And it could consist of anecdotal feedback as well. Questions can include how effective they think the training will be for them. Or what they would do with this new insight after the training.
Tier 4 – Knowledge. Testing for knowledge, understanding, and retention would take the form of a quiz. They could take a test at the end of the course to test for learning. Completing the test a few days after the course would assess for retention. This tests factual knowledge and could take the form of multiple-choice questions.
Tier 5 – Decision Making Competence. This part of the training evaluation could be done in many ways. For example, if part of the training is an on-demand course, the learner could complete scenarios about diversity situations as part of the online course. This could include selecting different options within the scenario and receiving appropriate feedback.
Tier 6 – Task Competence. A great way in this type, of course, to test for decision-making competence during the session is to use role-playing scenarios. For example, attendees could split into groups of 3. Two members of the group have a role-playing scenario to act out. While the third member observes the interactions and, using a job aide or reference sheet, provides feedback. After the course, team meetings can include role-playing to evaluate training effectiveness further.
Tier 7 – Transfer. An employee’s supervisor can do continued coaching and assessment and it can be included as a formal part of the evaluation of training. As well, there is also the option of completing an on-the-job employee 360 evaluation. This can provide a range of feedback on how the individual contributes to a positive work environment.
Tier 8 – Effects of Transfer. Employee engagement and satisfaction surveys can assess the longer-term effects of knowledge transfer. If there are control groups that did not take the training, a comparison can show the effectiveness of the training.
Thalheimer-based examples of training evaluations are more involved and take longer to plan and undertake. But they do provide a lot more in-depth assessment of the impacts of training and transfer of knowledge.
Thomas Gilbert (1927 – 1995) was a psychologist who used behavioral psychology to improve performance at work and school. Gilbert identified six variables that he believed were necessary to improve human performance: information, resources, incentives, knowledge, capacity, and motives. He called this his Behavioural Engineering Model (BEM). Gilbert believed that the absence of performance support at work was the most significant barrier to performance. And not an individual’s lack of knowledge or skill. So, he believed it necessary to focus on variables in the work environment before addressing them individually.
At the essence of the model, training is only one component needed to change behavior. And, to assess the success of the change, training evaluations have to be expanded to include these extra components.
At Spark + Co., we have adapted Gilbert’s original model to use in specific situations.
For this example of training evaluation, consider a situation in the Financial Services sector. A bank with a regulatory compliance problem often wants a simple solution and requests a training session. But we know that often this would very likely only make modest gains on its own. For instance, let’s use the example of a bank that needs to increase its compliance with Anti Money Laundering procedures.
To use this method effectively, you need to exert influence over more than just the training. Since you need to address other factors that impact the effectiveness of the process. To use this method, you will
All these factors contribute to an increase in performance. And while it’s more involved and needs more time to design and measure performance, it leads to enhanced results and improvement. And each of the five areas we use contributes to the progress. In comparison, training alone can only improve about 20% of performance on its own.
So, in examples of training evaluations of this kind, you need to expand the assessment outside of solely the training component. And test all areas to measure effectiveness and make future improvements.
Training evaluations for a course you are selling directly to a consumer are often quite different from the previous examples of training evaluations we’ve covered so far. And in many ways, they are much more straightforward. Let’s take the example of a yoga instructor selling a course on yoga techniques and practice. As well as developing evaluations for the skill you want your learner to perform, the following are less obvious aspects to include.
As with any course, you will want to assess the participants’ reactions and perceptions about the course. But the main question that you need to answer is how likely would the participant be to recommend your yoga course to a friend or colleague? This can determine your net promoter score (NPS). NPS measures a consumer’s experience and helps predict your business growth.
These training evaluations take the form of a question on a scale of 0-10. Those who score 9-10 are promoters. They are loyal enthusiasts who will buy your products and refer others. Thus increasing your growth. Those who score 7-8 are Passives. Although satisfied, they are unenthusiastic consumers. And are likely to be tempted by your competitors’ courses. People who score 0-6 are Detractors. And often unhappy customers who might damage your brand and slow growth through negative word of mouth.
By subtracting the percentage of Detractors from the percentage of Promoters, you get the net promoter score (NPS). It ranges from 0-100.
So in this example, your training evaluations could be one question. But we wouldn’t recommend that since you have an opportunity to collect a lot of valuable information besides your NPS.
For example, what about the price of your course? During the course development, you’ll have done market research to identify a suitable selling price. But are you leaving money on the table, or do your consumers feel that your course is overpriced? Asking a simple scaled survey question in your training evaluations can shed light on this.
You also have the opportunity in your survey to ask for a testimonial. This is often an excellent way to get testimonials to advertise and market your course. And if you have more classes, you can ask them their interest in taking one of those. Or, if you are considering creating new courses, you can provide them with a series of topics asking which they’d be most interested in—helping you gauge the market for your next course.
Customer Education is often much more challenging to complete training evaluations. That’s because you are evaluating three different sets of training expectations. Let’s take the situation where you provide training for a software product sold to enterprise customers. Imagine you sell project management software to other businesses. You want to sell the software, but you also want people to use and love your product. And your customers want their employees to be able to use your product, and they want to justify spending money on the software and its training. As a Customer Education manager, you are often juggling three stakeholders.
So in customer education, examples of training evaluations of each of these perspectives need to be included.
When creating the learner-focused part of training evaluations, you will want to assess their
But that’s not all. You also want to determine in the training evaluations the impact of the training. In this software training example, you want people to use the software. And many people feel that measuring the completion of the course is the critical metric. But that alone only has limited usefulness. It’s not much more than a Level 1 evaluation mentioned in the first example of training evaluations earlier.
Christy Hollingsworth (Senior Director Customer Education) at Heap agrees. Christy had this to say about designing training evaluations in a recent podcast.
“Over the years, I’ve started to care less and less about completions. You know, if they’re dropping off right away, I think that’s a very different problem. But if they’re getting 80%, through the content, and they’re not finishing the last few things, often, what I find is … they want to start applying it.”
So, measure activity and completion, but don’t focus only on that. Like with any other evaluation of training, you want to connect to application or transfer of learning. What would you see if your training was making an impact? How would you know? How would you measure that in your training evaluations?
As the “training provider,” you are also under pressure to evaluate the impact of training on YOUR business. As a Customer Education professional, when your boss asks, “is the training working?” you want to be able to determine:
As Christy H puts it,
“My favorite report is simply taking a course engagement, and then looking at the learning objectives for that, and then tying that to product behavior. So very specifically,…, what behaviors am I trying to drive with a particular course? And did I influence that in the product? When you can see that somebody hasn’t engaged with a feature or certain part of your product. They take the course … and then are now engaging … with that. And that is, I think, at the end of the day really going to tell you is your content successful at driving value.”
So in types of examples of training evaluations such as this you need to determine what your boss deems essential to the business:
Then based on what’s important at your company, you can establish the story that your training is trying to tell. For instance,
Our training offering will _________________________
So that ________________
Leading to ___________________________________
We will know we are successful when _______________________________
Then design training evaluation metrics to measure the level of success.
What does your customer care about when investing in your training? Usually, they need to know the time for their employees to reach competency with the software has decreased. And that they are receiving greater value because of the training. Your training evaluations need to measure these. This helps them know that they made a good purchase and that your training has a positive bottom-line impact.
Measuring this in your training evaluations can often be the most challenging. But if you can, measure account level “penetration” and active user data. At the “account level,” are enough of the customer’s employees taking advantage of the training? And, as we discussed earlier, you might use a survey as part of your evaluation of training to measure net promoter score.
For your enterprise customer (and for your boss) the training, improves adoption, product usage, and value-added. As well it improves customer retention and benefits the bottom line. So extra benefits of training evaluations are that they can show success both to your customers and your boss!
Although we’ve covered a lot so far in this article on examples of training evaluations, we’re not entirely done yet. Below are five extra tips and resources to help you create training evaluations.
Training evaluations are a simple concept, but to make them useful requires thoughtful planning. And as we’ve shown, there’s no one size fits all approach. The type of evaluation you use will depend on the circumstances. And by providing examples of training evaluations and the concepts behind them, you’re now better able to measure the impact of your training. And able to tap into of the benefits of training evaluations.
Spark + Co has worked with over 70 clients since 2007, providing custom online courses. We help businesses, non-profits, and government agencies to achieve their training goals. Need to create custom online training? Perfect, that’s what we do, and we’re ready to help. Book an initial consultation now, send us an email, or find out more from our website.