Update on progress

DDW Fieldnotes, 2/3/2012—Insight into Informal Evaluation

So, I had a real breakthrough today while swimming and spent awhile after swimming writing it down. Here is what I wrote on the iPad sitting in the locker room before going to the bike to read more in evaluation journals—

Most professional evaluators assume other stakeholders are poor evaluators.
(cite Stufflebeam, Fetterman, Scriven, others)
Similarly, most teachers believe they know more than their students.

What do evaluators do about this? Fetterman tries to help people do their own evaluations better. So does Patton in his own way. Others (Scriven and many traditionalists) call for better education with evaluation as a transdiscipline everyone should learn to do better but they focus mostly on helping professional evaluators do better evaluations for everyone.

Do we really know people are poor evaluators? Where are the phenomenologies or case studies or even serious narratives of people’s evaluation lives?

Where are the meta evaluations of what scholars seem to believe is just obvious?

They are out there but in a different literature than we're used to citing in evaluation theory literature. We need to look to the great literature of our novelists and to the biographies of people's lives to find many narratives of people's evaluation lives. We need to meta evaluate what these examples are teaching readers about good and bad evaluation.

In addition, we need to gather stories from our stakeholders about their evaluation lives if we hope to understand them and their values and standards. Or if we hope to make  credible meta evaluations of their evaluation lives, doing so might enable us to understand the real needs for education or how to help them do their own evaluations better and participate with us as their professional assistants or as their external evaluators who want them to take what we give them seriously.

So, this is the basis for a presentation of case studies of people's evaluation lives that I want to share in several different venues. In each I will describe the case's evaluation experiences and then critique them in terms of various standards, including their own, if they can state them.

Another key idea I had was that I don’t have to write all I learn about each case. I can simply tell a good representative story about each case, which will likely reveal many of the issues in a narrative format. Short stories will be easier to weave into articles, pull together into books, analyze for meaning, and I won’t feel as overwhelmed as I would if I thought I had to do a full evaluation biography on each person I included as a case.

These cases can then be used in a variety of ways- as descriptions of what it means to be human (kind of like Terkel and vanManen do), as fodder for evaluators to use as they reflect on their own stakeholders and how to work with them better, for evaluators to think about in terms of their own evaluation lives and what they could do to improve, for evaluation theorists to use in making better theories that take people's actual evaluation experiences into account, and for anyone interested in educating the young and all stakeholders to be better evaluators and to get more out of this life of evaluation experiences.

My personal view of the LDS religion and my Gospel basis for doing this work is simple- God put us here at our request to have experiences in which we would learn to distinguish between opposites and learn by making mistakes and repenting to choose the good, beautiful, effective, better, etc. By doing this, we have the privilege of becoming like God. Of course, we also have the possibility of rejecting God's offer  and losing all we have been given. We are agents and we have a choice; but if I can help people see that they are experiencing thousands of evaluation opportunities daily and they can be done better if they choose to do the better, I feel like I am helping do God's work of inviting His children to come unto Him and achieve joy through righteous use of agency to make good evaluations that impact them and others for this life and for eternity.

I elaborated on all this in a couple of audio recordings on the way to a meeting in SLC on Feb 3. I should transcribe those into here.

I also wrote this little entry based on the article I read while biking today-

http://aje.sagepub.com/content/16/1/59.full.pdf This article is old and works really hard to combine ideas from all approaches to argue that external experiments AND internal eval are both possible. But it fits with the AEA365 I read this morning that
talked about how to use qsort to get stakeholders to come up with common foci for an eval and the importance of engaging in the interests of all stakeholders.

It is focused on eval of outcomes within professional orgs and it still addresses many of the issues I've been wanting to address for daily eval lives.

I also elaborated on my case study of myself regarding weight loss efforts over the years in the audio recording I made while driving to SLC (after a short visit with Danny, so I’m really glad I have this phone!). As I did so, several issues about how I evaluate came up that I think are also relevant to most of my evaluations I make during this life of evaluation experiences: I am not just evaluating Fat Thermostat or alkaline dieting or veganism or green smoothie dieting.

Rather, I’m evaluating many of my personal characteristics as brought out by my interest in, use of, and eventual skipping of these diets. I’m evaluating not just my weight, but also my ability to stick with a plan, my tendency to be like the people around me (social eater), my habitual tendency to eat more than I need to, my true inability to make the evaluation that good health is more important than enjoying the tastes of good foods (I’m conflicted about good vs bad, since one is immediate and very appealing to my body and mind while the other is off into the future and not a sure outcome). I feel plenty of energy and have quite good health no matter what I eat or how much of it I eat.

So, I don’t feel motivated to change my habits as long as I’m enjoying what I’m eating. On the other hand, I really like it when people I like tell me I look good when I eat right. I also like it when the doctor tells me that my health risks are going down when I eat right (though he’s now saying it is mostly dependent on the metanex and chromium pincolonate that I take, more than what I eat or don’t eat).

Based on this, I’m thinking I should use Michelle Baron’s article to help her say what she wanted to say more simply. But I could also create five short stories of how these five evaluators she studied evaluate the needs of different stakeholders. Right now, I think our article should be saying something like this:

We don’t know much about how evaluators evaluate stakeholders’ values, especially when they are in conflict. Nor do we know much about how they do their evaluations in general. This study explores how five evaluators evaluate issues associated with the stakeholders they work with in various evaluations they have conducted for several years. Then, instead of giving all the details Michelle has in the article right now, I’d slim that back a lot to focus in on just a few key points to support the major points of the article.