A Critical Look at Measuring Learning Outcomes

A Critical Look at Measuring Learning Outcomes

Are you measuring the impact your learning and education programs are providing?

Hopefully your answer is Yes. Otherwise I must ask, "Why are you providing it?" Assuming you did answer Yes, then I must ask:

What are you measuring and why?

My question is a critical thinking question for you to contemplate about the specific learning outcomes ("At the end of this program you will be able to …") you are listing in your course design. Think about how they align with your organization's overall goal or the program sponsor's goal for measuring learning outcomes.

What do you and they hope the results will prove?

Expectations of learning outcome assessment results fall into four categories of quantification:

  1. Student performance

  2. Student competency

  3. Student growth in knowledge and skill

  4. Quality of the study program

Student performance

Typically, these assessments are formal tests, both written & oral, projects, simulations, etc. They are standalone assessments, are not formative evaluation quizzes, and results are not self-reported by the student.

Self-reporting and self-evaluation methods of performance assessments are not reliable and should not be used. Studies have shown that low performers overestimate their abilities and high performers underestimate them. (Dunning-Kruger, 2011)

If the assessed knowledge or skill is generic, not specific to a job role or application, then assessments of this type do not directly correlate to performance levels on the job. In other words, the results reveal whether they passed the controlled test to the degree specified in the behavioral objective(s) of the study program, not whether the student can specifically apply a knowledge or skill in unique job situations.

Student competency

These types of assessments correlate directly to specific jobs, roles, technologies, etc. They typically use more complex assessment methods such as independent projects. Group projects, where the students each contribute as part of a team, are much more difficult to use to assess an individual student's competency.

The determination of competency cannot be made from a single point-in-time performance assessment, it requires multiple assessments with varying levels of complexity. Multiple observers/assessors are also usually required.

Competency assessments group a set of individual behavioral outcomes, covered in one or more study programs, into a single definable ability. For example, the competency of Problem Solving, defined by 3 to 5 observable behaviors, is the focus of the competency assessment.

Student growth in knowledge and skill

Pre- and post-testing of the knowledge and skill study content is a requirement for assessments of this nature.

For accuracy, objectivity, and ease of scoring reasons, they are most often only written assessments with multiple choice type questions that test knowledge acquisition.

However, if highly a structured interview or observation method is employed, they can also successfully measure a student's skill ability at adhering to specific processes and procedures.

Quality of the study program

In corporate training and development sponsored environments these data, often captured from what is known as "smile sheets," are self-reported measures and do not measure actual outcomes. They only measure students' feelings and attitudes.

Any self-reported or self-evaluation methods do not measure actual outcomes. That is important to understand in programs that claim to follow the principles of Adult Learning. Self-evaluation is an important tenet of adult learning, but remember, low performers tend to overestimate their achievement and high performers tend to underestimate it… thus it is an unreliable measure of current or future performance.

In all generic commercial learning venues, the reported behavioral learning outcomes, (i.e. "At the end of this program you will be able to …") fall into this category of measurement. From an actual assessment of learning standpoint, all you can really hope to measure is if the student attended/took part to some degree and did they subjectively find the time valuable.

From a Critical Thinking standpoint, understanding why you are doing something is always important.

If you think about that from the standpoint of the student you are trying to affect and the other stakeholders in the process, that should help you prove your designed structure for learning is producing the right outcomes.


Quantity is More Important than Quality

Quantity is More Important than Quality

Inserting Images in the Age of Privacy

Inserting Images in the Age of Privacy