Beyond the Test: Designing Performance-Based Certification

Before embarking on the design of a performance-based certification program, it's essential to consider key steps to prevent common mistakes. Our guest, Judy Hale, a seasoned speaker, author, and principal at Hale Associates, delves into this topic in her acclaimed book, "Performance Based Certification: How to Design a Valid, Defensible, Cost-Effective Program." 

Show Notes: 

Guest Judy Hale offers many actionable steps to take when designing a certification program. 

• Certification is a Means, Not an End: Judy emphasizes that certification should be seen as a vehicle to achieve specific outcomes, rather than just a goal . Organizations need to clearly define the problem they are solving and the desired outcomes before designing a certification program. 
• The Importance of a Thorough Job Analysis: A key step in creating a defensible certification program is conducting a comprehensive job analysis. This helps ensure the certification is aligned with actual job requirements and includes input from all relevant stakeholders. 
• Customization is Crucial: Certification programs must be tailored to the specific needs of the industry and the target audience. Judy highlights the differences in certification requirements across various industries, illustrating the necessity of customization. 
• Leveraging Technology and Best Practices: Judy discusses the increasing role of technology, including AI and virtual reality, in developing and administering certification programs. Additionally, she emphasizes best practices like ensuring cost-feasibility and focusing on performance-based assessments to create effective and sustainable certification systems.

Learn more about Judy Hale 

Judy Hale’s Books 

Powered by Learning earned Awards of Distinction in the Podcast/Audio and Business Podcast categories from The Communicator Awards and a Gold and Silver Davey Award. The podcast is also named to Feedspot's Top 40 L&D podcasts and Training Industry’s Ultimate L&D Podcast Guide.

Learn more about d'Vinci at www.dvinci.com.
Follow us on LinkedIn
Like us on Facebook

Transcript

Susan Cort: [00:00:00] Before you design a certification program, there are many questions you should ask to get better outcomes for your organization and your learners. 

Judy Hale: Since most people tend to think of tests, I keep saying, so what's the problem you're solving? If everyone passed your test, what miracle will happen in this world?

Judy Hale: What will be different? And I find it, one of the mistakes is, people talk about we want to certify, that's a vehicle, that's not a goal. You assume that the certification will produce an outcome for you. That's 

Susan Cort: not a goal. That's today's guest, Judy Hale. An expert on performance-based certification, Judy will step us through how to design a valid, defensible, cost-effective program.

Susan Cort: Next on Powered by Learning. 

Announcer: Powered by Learning is brought to you by d'Vinci Interactive. d'Vinci's approach to learning is grounded in 30 years of innovation and expertise. We use proven strategies and leading technology [00:01:00] to develop solutions that power learners to improve quality and boost performance.

Announcer: Learn more at dvinci. com. Joining me today 

Susan Cort: is Jenny Fedullo, d'Vinci's director, learning experience, and our guest, Judy Hale. Judy is a longtime speaker and author, and is the principal at Hale Associates. She joins us from her office in Illinois to talk about her book, Performance Based Certification, How to Design a Valid, Defensible, Cost-Effective Program.

Susan Cort: Hello Judy! 

Jenny Fedullo: Hi, 

Judy Hale: Judy. It's nice to see you again. Well, thank you both for having me here. I'm looking forward to answering your questions and finding out what your perspectives are. 

Susan Cort: Well, thanks, Judy. Let's, let's start out by giving our listeners a brief overview of your career so they understand kind of the experiences you're bringing to today's conversation.

Judy Hale: Well, um, I started my company in 1974. And over the years, [00:02:00] I have mostly been hired to design or audit or validate credentials. Now what I mean by that are certifications and certificate programs. Now when we started those many years ago, most of those were offered by Trade and professional associations.

Judy Hale: Now the larger player are corporations who want to certify their, not just their own employees, but their suppliers, their aftermarket partners. So many, so many companies have outsourced. So they, they want to make sure that the groups that they've outsourced really carry their brand, represent them, are competent in that way.

Judy Hale: So, I have worked across all industries, uh, by the way, the most amusing, best ones are amusement parks, and I've worked with medical, the high tech people, and, uh, I've worked across both private and public sectors. [00:03:00] Does that help you a little bit? You need to know more? Yeah. 

Susan Cort: No, that's great. That's great.

Susan Cort: We'll dive right in then. 

Jenny Fedullo: Yeah, it's quite a niche that you found for sure. I know that we've worked with several of our clients to certify their courses and you don't see much out there to help guide companies. So when I came across your book, I was, I was pleasantly surprised to see that it aligns in many regards with, with the approach that we take.

Jenny Fedullo: I'm curious, Judy, what, what inspired you to write it? What, what, um, What was the catalyst, and how did you identify the critical components that were gonna, that make a certification program valid? 

Judy Hale: Well, you are the right on example of why I wrote it. People want to do this work, but there was no easy resource.

Judy Hale: Oh, sure, you can download ISO ANSI standards, or any of the other credentialing or crediting bodies. But it doesn't walk you through the process. And I, I saw lots of good people doing their best without guidance. So I went to the associations that were [00:04:00] doing this, but they were unwilling or uninterested.

Judy Hale: So I said, why not? So I actually used groups that are, have very good programs, and I interviewed them and, and they provided the basis behind that. There was nothing in the marketplace if you wanted to do this. So why not provide that? Now you could go to annual conferences, but that's a rather inefficient way to learn it.

Judy Hale: Go once a year for four days and hopefully you went to the right sessions. So it seems like, again, it was a very inefficient way. But I saw good people struggling and it made, that made no sense to me. So I had the audacity to write a book and that book has actually been very well received. It's won awards and I've been told by people how much has been very useful for them.

Judy Hale: Okay. But I, I didn't think it up by myself, please. I actually work with clients that have very good, very robust programs. What they learned and they so many times when you think about certification, they'll talk about the [00:05:00] test. Mm hmm. And even though that's important, it's a very small component of the whole process.

Judy Hale: Okay. 

Jenny Fedullo: Yeah. Yeah, absolutely. Curious, you know, at a high level, we will you walk us through the step by step process that you recommend and and maybe what are some common pitfalls that that organizations should avoid? Well, certainly, 

Judy Hale: since most people tend to think of tests, I keep saying, so what's the problem you're solving?

Judy Hale: If everyone passed your test, what miracle will happen in this world? What will be different? And I find it, one of the mistakes is people talk about we want to certify. That's a vehicle, that's not a goal. You assume that the certification will produce an outcome for you. So the goal, then the first step is really what are we talking about?

Judy Hale: What is the problem? What are the metrics? And by the way, who, who are these people that we want to certify? Are we talking about members of our profession? Are we talking about people who completed our training program, [00:06:00] academic program? Who are we talking about? And again, it's amazing how people talk like they're in agreement, but when you get them to define those attributes, you find how, how apart they are, actually are, okay?

Judy Hale: Then the next step is, okay, now that we've agreed on why, who, and what will track measures, now let's do a job setting. Now this is the piece that's key to defensibility. So in a professional trade association, you go out and you actually tend to survey those people. But in a corporate world, it's a closed system.

Judy Hale: You're not asking about how Harry did it. You're talking about your proprietary information, your proprietary rules, but you have to define those, and here's where the mistakes are. You don't bring in stakeholder voices. You bring in the person who designed it, not necessarily the people who depend on it.

Judy Hale: Now what's this going to look like? Are we going to allow a pre test for people to see how ready they [00:07:00] are? Are we going to provide training, resources? Are we going to give people feedback? What's this going to look like that way? Once you've got that, and someone in parallel, you might ask, who's going to oversee this?

Judy Hale: So in a corporate world, it would be someone in HR, maybe the line organization depending on it, and maybe some other vested parties. Powered by Learning. They are key to define, well, what happens when a person fails? Do we allow them to retake? How many times do they get to retake the test? And if it's a corporate world, do they get to study on the clock or on their own time?

Judy Hale: And how many times can they retake the test? In a professional trade association, they have questions like, is the test on demand or is it only tested given once a year? Do you have to wait a whole year to do a retake? How do you do those things? So you need a body, not just the training organization, or [00:08:00] you actually need an oversight body, and you need to make sure it's representative of the stakeholders, and those are the kinds of policy decisions that they're going to incorporate for you, okay?

Judy Hale: Once you've done that, now you actually go out and build it. So you Build the test, build the reference, the training, the marketing piece, how you're going to promote it, sell it, whatever, and things like that. In that building, you should be doing formative tests, beta tests. Does it work? Do you have the infrastructure?

Judy Hale: How are you going to deliver this test? Do they have to go to a test center? Can they do it at their desk? So it's all that administrative. A lot of people don't think through. And then, of course, you implement and you track and evaluate. Are the miracles happening that you wondered? Are people signing up?

Judy Hale: So any, anyway. One of the mistakes is that groups in their desire to be elegant will create something that's actually administratively so burdensome, too [00:09:00] costly to administer. So these are the kinds of decisions you need to think through. So we built this horse. Can we really ride it? How long will it go?

Judy Hale: I'm sorry, I'm big on analogies, so I can go overboard. I hope I'm getting the point across. That you really need to think through how this is going to look when it becomes live. Well, if those of you in training know, how are you going to keep your content current? We build training. We don't think through how agile our systems are.

Judy Hale: How can we update the content? 

Jenny Fedullo: Right? I think those listening would feel reassured that the steps you just walk through should be familiar to, to those listening, right? These are the steps. Steps we go through and we create a learning solution, right? We start with what's the problem? You know, we do a test analysis.

Jenny Fedullo: We then move to design, then development, then implementation. So I, I think it's probably reassuring for those listening that it's, it's a familiar process. Well, again, too many people start with building the test. Exactly. Yep. They start with the solution at the end. Yep. So, moving on, [00:10:00] you highlight the idea that to expect performance from employees, organizations, they have to be equipped with the right tools and resources.

Jenny Fedullo: Could you elaborate on how this principle impacts the design and effectiveness of a certification program? And perhaps maybe, maybe share some, uh, best practices. 

Judy Hale: Let's talk about you want to do this. Where are these people you want to certify? It's one thing if they work in your office and they're down the hall.

Judy Hale: It's another thing if they're on all six continents around the world. Learn It's another thing if they all share the same background. They all went to the same kind of schools. But when you think about it, a lot of people, their point of entry is very diverse. Certifications were originally designed for people who completed accredited academic programs.

Judy Hale: That's a small universe anymore. If you think about roofers, I do a lot of work with them. Well, how do you get to be a roofer? My heavens, your point of entry could be any number of things. And by the way, there are 15 different roofing systems. And [00:11:00] by the way, knowing one doesn't mean you can do another one.

Judy Hale: They're so unique. So, part of that is, how are we going to administer this? Do we even have a database system to track who applies? Do we have a system that will administer a test? Is it going to be a hands on test? Oh my heavens, when are we going to do that? The roofers, because they're a unique situation, because their warranties are by the manufacturer.

Judy Hale: So the manufacturers have a need for installers to be competent. Because a mistake by the installer, the installer doesn't get punished. The manufacturer gets punished. Their warranty is violated, see. So what happens then, so, what happens How do they do that? And by the way, roofers don't do multiple choice work.

Judy Hale: None of us go to work and answer multiple choice questions. Multiple choice are so removed from the work, but it's the primary way in which we test. And they require a codified body of knowledge. Have we ever bothered to [00:12:00] codify? Which books are we going to talk about? Whose manuals? Who wrote them? How old are they?

Judy Hale: Are they still current? So, that's the infrastructure. Am I answering your question? That's the infrastructure. 

Jenny Fedullo: Yeah, yeah, no, I love, I love the specific examples about the roofers, right? That, that really helps put it in context. 

Susan Cort: Yeah, that's helpful, Judy. How about some advice for people just getting started with the certification program?

Jenny Fedullo: You know, pick three best practices that, that those listening, you know, would be super helpful for them when they're, when they're faced with this challenge. Okay, I've 

Judy Hale: got 10 best practices, but I'll give you three. One of them is actually designing a system. That is cost feasible to administer. Another best practice is try to be performance proficiency based.

Judy Hale: Now this is how we can do through portfolios. That's the new areas. Project based certification. You're given a project, you use criteria, and you deliver. Or portfolios, you actually produce copies or samples of your work. Those are the really new ones coming up. And another [00:13:00] best practice is really to take the time to correlate the number of people who completed with your miracle you thought was going to happen.

Judy Hale: Take the time to say, is it, does it matter? Are, are fewer accidents happening? I, I am not impressed by high failure rates. That just tells me you have not adequately communicated the requirements. Shame on you. So don't, don't tell me about. Oh, only 30 percent of our people pass. This is something you should be ashamed of, not proud of, because you have not done the job of adequately setting the expectations and helping people get prepared for what they're doing.

Jenny Fedullo: Yeah, that's fair. Yeah, we shouldn't be setting our audience up for failure, right? We should be building it so that, yeah, the right way so they pass, exactly. Right. Yeah. So your, your book emphasizes, and you talked a little bit about the importance of, of customizing the programs with the, with the roofers.

Jenny Fedullo: Can you share maybe some other examples of [00:14:00] how different industries can tailor the programs to meet their specific needs? 

Judy Hale: Part of this is we talk about doing a job task analysis. I think we need to do a practice analysis. Now I'll give you the right on examples. For example, dentists. Let's talk about dentists.

Judy Hale: Dentists are board certified, but a dentist in the United States is different than a dentist in other parts of the world. In the United States, you have to know how to administer antibiotics to prevent infection. You have to know how to prevent pain. In other countries, dentists don't do that at all.

Judy Hale: Somebody else does that. They just drill and fill. So, customization is looking at what's different in practice based on legal changes around the world, based on access to resources around the world. So, we tend to think that a job is the same around the world. It is not. In some parts of the [00:15:00] world, you have access to resources.

Judy Hale: Other parts of the world, you don't. Yeah, 

Jenny Fedullo: absolutely. Learn So, with so much going on, rapid advancement of technology, AI, changes in the workforce, how do you see the role of certification programs evolving in maybe the next, you know, three to five to ten, ten years even? There's actually, two arenas. One is how the certifying group uses technology to create its own tests and administer.

Judy Hale: So one, we certainly have AI being used to generate questions. That's very real. But what about virtual reality to use to actually assess? Virtual reality is now much more cost competitive. The equipment is smaller, it's much more cost feasible. You can actually test people in very discrete medical procedures.

Judy Hale: The trades are using it to teach safety and they're actually assessing people in teamwork. So virtual reality is a technology [00:16:00] as well as AI. That we use for the creation and the delivery of our tests or certifications. On the other side, you have the person you want to certify and how is technology affecting their job.

Judy Hale: And now we have what we call badges. So I might certify you in this small aspect, which is maybe using a software, maybe it's a shared drive or electronic tool, I don't know. So well, vendors, this is, this is one. Vendors. They're going to produce software and they now certify their customers in the use of that software.

Judy Hale: So if you had a software that enabled the legal profession to track all the venues of cases, you would want your administrative assistant to know how to properly pull that database and generate reports. So they actually certify their people to do that. So now you're certifying people in the use of a technology, search [00:17:00] engines or whatever, in doing their work.

Judy Hale: Now maybe, that may only be a small piece of the work. So that's where badges come in. You can certify them in the use of this software or hardware and as, and then maybe you certify them in other things. One of the new phenomena is what we call the creation of badges or smaller micro credentials. 

Jenny Fedullo: Yeah, I know we, we certainly use badging, um.

Jenny Fedullo: As a way to recognize learners, you know, when they've completed certain things or they're they've, um, you know, to show that achievement. So it's good to see that correlation for sure. Thank you for opening us up to this world of certification, right? Yes. Many listening probably didn't think much beyond, you know, a test, right?

Jenny Fedullo: So it was really interesting. Yeah. Thank you. 

Susan Cort: I agree. Yeah. Thank you, Judy. This was really interesting to learn. And not just the step-by-step process for designing a certification program, but hearing a little bit about the pitfalls [00:18:00] and the importance of customization and all the ways that technology, uh, is impacting this field.

Susan Cort: And certainly you've seen it all over the years. So we want to thank you for joining us as our guest and sharing your insights with us today. 

Judy Hale: Well, thank you for having me. People get some insights from what I was able to share. 

Susan Cort: They will. Thanks, Judy. 

Judy Hale: Thanks, Judy.

Susan Cort: Jenny, Judy's points were really interesting for our listeners wanting to create performance-based certification, but you know, we don't often hear much about the process for making that happen. 

Jenny Fedullo: I agree, Susan. It was such an insightful discussion. I mean, to have her book and the process she laid out. It's so valuable for organizations aiming to create those defensible certification programs.

Jenny Fedullo: And her step-by-step guide really emphasizes the importance of approaching certifications just like we approach creating custom learning solutions. 

Susan Cort: Absolutely. Thanks, Jenny. And special thanks to our guest, Judy Hale. If you have an idea for a [00:19:00] topic or a guest, please reach out to us at poweredbylearningatdvinci.

Susan Cort: com. And don't forget that you can subscribe to Powered by Learning wherever you listen to your podcast.