How do you measure effectiveness when transitioning from presentation of content to a constructivist-based, distributed learning model?

Hi Janet,
I did my Master’s thesis on “Evaluating the Effectiveness of Non-formal Learning (NFL) on Knowledge Workers.” The research is based upon practices and processes I use in my job as a learning & development professional.
I’d be happy to discuss the theory and practical application of my approach, and the methodology I use with you (e-mail address supplied).
Best regards,
Michael Hanley

Evaluation of a course, module, or learning experience doesn’t strike me as being quite the same as an assessment or measure.

Think of the following terms:
Objective
Aim
Goal

Each of these terms have the same purpose. To define a specific learning target and possibly a predefined degree of proficiency. The only real difference in either of these terms is from whom shall we base perspective? The instructor? The learner? Some established mutual understanding of a team?

The French may say, “The uncooperative book, she fell from my hand.”
While the English might say, “I became clumsy and dropped my book.”

One culture places the emphasis on the world at large and how it interacts with humans, while the other places it on the individual human, and how he interacts with the world.

The bottom line of both sentences is the same. The book is no longer under the subject’s control.

Learning Theories often have us arguing in circles about how to word the same universal questions answers……

For a good assessment, perspectives such as these are not so important. We simply need to know if the learning target has been met, and to what degree of proficiency. In the case of assessment, it is critical to have a goal, aim, or objective as well as some standard or rubric from which to judge or measure results. From an assessment point of view, does it really matter which ‘perspective language’ is used? We just want to know if the learner can do, and how well he can do.

In the case of assessments, there indeed must be some pool or pre-conceived set of ideals, standards, and skill or knowledge sets to work from. Even in a very ‘free thinking’ user created simulation such as Second Life, there are quests for the learner to choose from (or be assigned). “Rez a 10×10 cube, texture it, color it.” “Discover some of the many methods to create or obtain clothing for the avitar and wear them.” “Join or create an interest group.”

Evaluation is a whole different animal. This is when the different perspectives are open for debate. Evaluations examine over all learner experience from a variety of ‘perspectives’. To me, evaluation should be a far more ‘qualitative’ process when at all possible, and is usually more about developing learning experiences that people ‘like’ and will ‘choose over the competition if given a choice’ than it is about sheer effectiveness and accuracy.

Kirkpatrick offers a very good model to start with (Reactions, Learning, Transfer, Results).
From there, expand the model according to evaluation needs.

This basic evaluation model stresses the importance of the ‘old fashioned ideas of learning targets and clear assessment standards’; but, it also opens the doors to examine ‘perspectives’ and ‘hidden curriculum’ possibilities.

Is there room for more research on ‘evaluation’ models? Absolutely! A good deal of evaluation is usually done ‘after the fact’. Perhaps experiments in rapid prototype designs will show there are interesting ways to evaluate in real time, without causing designers to loose sight of those important measureable ‘objectives, aims, or goals’.

I fully agree with you that evaluation is important. How ‘WE” view learning.

I do not agree that theories are not a mix and match thing to a large degree…and that WE always fit in one or the other. Even the most astute of educational institutions have evidence of mixed learning theory approaches, wording, and practices. I.E. Portions of a state’s benchmarks developed by one group of people, from a different ‘era’ might use behavioral terms such as ‘behavioral objective’, while another section, amended some time later by different people picks up on more cognitive terms like ‘aim’.

Behaviorism – Observes the learner from an external point of view. The subject could not do this before. We gave him a solution and trained him to complete a task. Now he can do it. Success!

Cognitive Theories – Attempt to observe learning through a system of consistent ‘models’, and accumulative data on how the individual brain works. The subject could not do this task before at age 6 after a specific activity in environment A. Upon being moved to environment B, using the same activity, the subject was instantly able to complete the task. Other things effect learning besides repetition and reinforcers, so we need to consider them very closely. Success!

Constructivist Theories – Combine the former two, and draw into the equation models for more complex observations involving groups, collaboration, clustering, informal learning environment and methods, and the list goes on.

All three are looking at learning from different perspectives, and asking the questions that are not accounted for in the other existing theories.

Mixing and matching is imperative.
Even so, one can still see familiar processes in all of the theories. They just get new names, and different ‘points of observation’. Chaining and Training by a few select individuals vs. Scaffolding like models through the observations of many…including the learner himself.

Quantum and Relativity theories of physics do debate, but without both perspectives from which to view the same phenomenon, many of our latest world technologies would be grounded. In fact, there is much activity searching for the missing link that welds the two together (The Theory of Everything).

Evaluation of learning and curriculum, as I understand it, still is not really the same thing as learning assessment.

The process becomes more QUALITATIVE in the sense of expression here. We transcend through the various combinations of procedural, descriptive, conceptual, and critical forms of evaluation to arrive at ‘active theorist’ solutions which are often unique to direct stakeholder preferences. It’s very hard to put ‘numerical measurements’ on values, emotions, and schools of thought. What flies in Harlem, NY, just might bomb in Redmond, WA, and vice verse. Yes, we can create statistics and study the trends, but the scale and conclusions will always change depending on the nature of the evaluation.

The key is in having some idea of the specifics one wishes to examine in an evaluation.
Once this is determined, go mine the data and study it.

One thing technology offers us today, is the ability to meld the gathering of evaluation data in with those more directly related to daily learning objectives/aims/goals. It is not as expensive and time consuming to collect views on learning from more sets of stakeholders for ongoing or later study, WHILE they are learning, as opposed to older methods of ‘after the fact’ evaluation.

Other ideas are more validity tests for evaluation data that comes in the form of ‘objective’ style surveys. I.E. How different are the results of a single survey filled out by a group of individuals who collaborate and vote on each answer, as opposed to that same survey filled out by individuals in private cubes with a curtain drawn? What can we learn by comparing the two? Do any obvious patterns emerge? What do learners say in an essay about a learning activity, as compared to a questioneer about the same experience? What happens if we ‘interview’ learners, take an essay, and do a questioneer; then, compare the three? Are they consistent across all three formats for communication?

Very interesting stuff to think about as usual. Thanks for sharing it all in your blog.

Thanks Janet,
No rush…life is busy I wish the best in your activities and goals. Sorry if I get wordy at times. I enjoy blogging, and your topics and mediating threads are quite thought provoking.

On learning theories, I’d like to clarify that I did not mean all of the theories are the same, just with different wording. What I did mean to imply, is they OFTEN do send us debating in circles over how to word the same sets of observations, as they also aruge over who should be making the observation.

The main thing I wanted to point out is the strong difference between ‘evaluations’ and ‘assessments’.

In an evaluation, everything matters. We look at everything from theory, to cultural beliefs, to how full a child’s belly is at school.

In a good learning assessment, none of that counts. How the learner learns, what his background is, etc…is a mute point. Does he know it? Can he do it through this particular assessment instrument? How well?

I do agree that there are alot of ‘bad’ assessments out there. I also agree that more thought should be put behind assessment instruments and styles. They do not always measure the right things at the right times.

Evaluations help us learn to make better (or more appropriate) courses and assessments.

Brian