How I evaluate differently in my daily life versus in my teaching life versus in my professional evaluator life?

I wrote the following as a kind of self-interview as I was exploring the differences in my evaluation life as a professional evaluator, as a teacher, and as a family member. I wanted to think about how my views on values, evaluands, stakeholders, etc vary across those settings. I also wanted to explore how these issues might be of importance in understanding others' evaluation lives. These notes were included in Gong's materials on my evaluation life.

DDW Fieldnotes--Three Categories of My Evaluation Life

Started 9/30/2011 and completed 10/14/2011

The thing I want to dictate is my thoughts about my self-evaluation or how I evaluate. I do more than self-evaluation although that's one of the things I do evaluate.

So, I think one of the 1st things I need to talk about is to distinguish between how I think about evaluation as a field, what I'm trying to learn and say about it as a researcher and as an evaluation professional, compared to how I evaluate in my daily life. I think those are distinct things; but I would like to find out.

I think the way to find out will involve describing several evaluations that I'm involved in in my life and then look at them to see if they are different when I'm evaluating officially from when I am not.

1. So my 1st example is the evaluation I'm doing right now with Randy Davies for the Lumina foundation for the Utah group of universities regarding Tuning. I'm going up right now to observe in Salt Lake at a majors meeting. The majors meeting, as I understand it, has been going on among people who are teaching the same major at different universities in Utah for the last 10- 20 years. So, as I'm thinking about it, this is a part of the phenomenon that I'm supposed to be helping evaluate that's already been in place; and I'm supposed to be learning the nature of it in preparation for helping evaluate it.

If I stop and think about it in terms of formal evaluation language, I would say that I’m going to meet some of the stakeholders and to get a feel for the relationships among these people who get together and talk regularly about their majors, how they teach their majors, and what they expect of their students. This is all part of the evaluation that the Lumina foundation and Tuning project are trying to tune into.

Why else would I go up and see this other than to try and understand the phenomenon better from an evaluator perspective or from the perspectives of multiple stakeholders? These multiple stakeholders seem to me to be the faculty from each of the universities, representing the interests of the faculty at home who aren't able to come to the majors meetings. They also represent the interests of the students they teach.

So again as a formal evaluator I'm going up to Salt Lake to listen in on the majors meetings, anticipating that will allow me to better understand the stakeholders, their values and the nature of their (the values) value from the perspective of those who participated, again using informal evaluation language. With all  that I'm supposed to help representatives of the same people evaluate their attempts to come to a greater consensus on what they're supposed to be doing in terms of student learning outcomes.

As I'm going to this meeting, I am thinking that I've had about 2 days worth of meetings and interactions with several people associated with the Tuning project. I've heard their statements about the status of their efforts. And I'm pretty sure I know that they have been mostly establishing good relationships with each other, trying to figure out what assessment means, what they are going to be able to do to assess the learning outcomes. And they are coming up with several other agenda items that they're working on. Some of the questions that I have in my mind are: hasn't this already been done through their professional organizations they represent, for example, physics and history, psychology, elementary education and other disciplines that might come on board later, mathematics for example.

However I'm also thinking that people would not be spending as much money and time and effort if these things had already been done. Or they want to do them themselves so that they can basically get everyone on board anticipating Tuning as being a radical issue that they might HAVE to address someday.

So I'm going in open-minded and willing to be informed and taught about things I don't necessarily understand yet. I'm also being paid a pretty handsome sum for participating in this that makes this kind of evaluation different than the normal everyday evaluations I do all the time.

It might be interesting to explore how being paid makes a difference from not being paid. Some of things I think that makes a difference for me are:

1. I am responsible for doing a good job beyond my own self interests and beyond just what it means to be responsible and helpful to fellow beings. I wonder if this is a legitimate reason for being paid and also if my reasons for doing a better job are legitimate because it seems that I ought to be doing a morally good job or at least my best whether I'm paid or not. Yet being paid does seem to add something.

2. I think another difference between doing formal evaluation and my normal everyday evaluations is that I've been trained as an evaluator and have been teaching people about professional evaluation for 30 years. I have had to defend what I do to the people who taught me and to others that I've been doing studies for and I've used the standards that the joint committee on evaluation have come up with. So whenever I think about doing a professional evaluation like this one I think about those standards and how well I'm meeting them. I don't think about that all the time but I do think about it regularly.

When I do I usually think I'm doing pretty well. But I also think I better evaluate myself responsibly. But maybe I wouldn't think I’m doing so well if I were to systematically look through those very standards (which I don't do on a regular basis because it seems like too big of a job to go look at all those standards every time I make any evaluations.

So getting specific as I'm going up right now to observe and listen in on the majors meetings, I think that I'm doing a pretty good job as far as my meta-evaluation standards go. Why? Because I am attending to the peoples’ standards and their understanding of what is important. I'm trying to get into their heads to understand the experiences of the stakeholders. I like doing that because of my interest in qualitative research, which has always been kind of at the core of my interest in evaluation. But I also know that this is meeting a standard to try and understand the values and perspectives of local stakeholders.

I'm also going and doing this because Randy and I agreed that we should and we were invited to do so by Janice and the team members she’s working with. So, I’m being responsive to some of the key stakeholders that I think are invested or involved with this project.

3. Another way that being paid influences me is that I'm willing to come back and attend to issues over and over lots of different ways rather than just see the evaluation that I'm doing as being a one-time opportunity. So for example I went to a meeting last week, two different meetings last week actually. On Tuesday we met with the team from the Midwest where we were talking and listening to them talk about their values and the issues and concerns that they had. Then we met with the physics group and heard from people from around the state about their values and issues and concerns. And now I'm going up to the majors meeting to hear people who aren't directly involved with the tuning project but who are nevertheless some of the stakeholder group. You might think of this as a form of triangulation and it's a very systematic thoughtful approach that has all kinds of evidence to support doing it.

4. In our meetings we've been talking to people we’re working with about the approach that we're going to take. Having an approach seems to be something that's unique to doing a formal valuation. Whether I actually use a single formal approach or just talk about it are two different issues; but when I'm doing a daily regular everyday evaluation I don't think I ever would think about any formal approaches. In this case we talked about using developmental evaluation; and we've also thrown in the terms formative and summative. I think we've used those terms because that's something that these stakeholders are familiar with and they want to have something of value and these terms at least sound like we’ve got something substantial in mind for them to consider. We can defend what we’re charging them money for by citing Patton and Scriven and others. Although we do plan to do something in these three areas, we are mostly trying to be responsive to stakeholders and their wishes. That isn’t something I usually think about when doing my informal daily evaluations—who the stakeholders are and how they might feel about things and me trying to be responsive to them.

During our interview on Oct 13, Andy Gibbons expressed well what I’m thinking about now as I’m reading and editing this self-evaluation analysis. He said that thinking about stakeholders, evaluands, criteria and methods is very rational and systematic but not what he does when he evaluates. He says lots of things play into his evaluations, which he has displayed on a paper he gave me and which I need to get an electronic version of (see it in my interview of Andy which is in the “named” or “anonymous” folder). But he chooses among all those in making decisions in a complicated way that is driven mostly by the emotions he feels when he has evaluation choices before him. I think I probably do the same thing, even though I am reticent to admit it because I want to be able to tell other people who are paying me to do formal evaluations or to teach them how to do them that there is more to my evaluation work that is rational than just emotional.

5. I might go on thinking of things about this project that are different. For example, we’re going to have a responsibility of formally analyzing what we're learning, which means we need to get some formal record of the data we’re gathering. In order to do that we are going to keep a running record of what we’re learning and finding ways to share that with Janice and the team. So this will have an emphasis on formal reporting and analysis. Sharing requires a lot of work and effort that we don't take the time to do in regular daily evaluation.

B. So now maybe I could briefly think about evaluation that I do as part of my formal work that is not specifically formal evaluation studies like the one I’ve been talking about. For example one of the main things I do is teach. Right now I'm teaching 4 classes: the Book of Mormon, the MUR class, advanced qualitative research, and advanced evaluation.

So in these classes what evaluation am I doing?

1. One of the 1st things that comes to mind is I'm evaluating how well I am helping the students. Thinking in particular about the Book of Mormon, last night I decided to send, at the last minute before I was going to bed, an e-mail to some of the students who are not doing as well as the others in terms of the number of points they earned so far. In writing that e-mail I basically was going off of the experience I had earlier in the day when I met with 2 of the students and they were both very appreciative that I have invited them to make appointments and we sat down and talked about how many points they had wanted versus how many they have so far, and what we could do about it. So I decided maybe I should invite some of the other students who were getting less than they might now. I invited them to meet with me or to at least realize that they could make up some of the points and that they shouldn't feel penalized and give up on trying to do anything more with the class than they are already doing.

So in this situation I feel like I was evaluating the feedback that I've given to the students, the grades that I've been giving to them, how clear the syllabus was or was not, and my relationship with the students. I find that I want them to like me, I want them to feel at the end of the class that they were given a fair shake, and they learned. Related to these points I am regularly evaluating how they respond to me during class as well as over time when they talk to me about how they feel about the class and about their learning in the book of Mormon.

I reflect back on how I felt about Bruce Hafen when he was my teacher and how much I valued his point of view and him as a human being. I hope my students feel at least somewhat as positive toward me because I feel like my interest in the Book of Mormon went up because of my interest in his life and in the stories he told and the example that he was to me. I don't imagine that my students will feel that way about me; but I'm hoping that they at least won't feel negatively about me and may want to move in that direction.

2. Another kind of evaluation I’m doing of myself as a teacher is whether I'm covering material the way it ought to be covered. In the BOM class I know that there is so much more there than I can possibly share with the students. Because of that evaluation I've made lots of choices focused on having the students learn how to learn and be excited about exploring resources on their own with the hope that in the long run they will be continuously learning and growing and exploring alternative resources throughout the rest of their lives when they have nothing to do with me or this class.

Also I’ve emphasized teaching some key ideas such as learning by teaching and sharing, learning by doing, learning through service, and using class time for discussion so that the students are doing the teaching and raising their hands so that the Spirit can teach them more profoundly (see talk by Elder Scott on this). Having all of those changes in how I teach have come about because I constantly ask myself if the students are learning what they should learn in this class and what I can do to help them learn better.

I use a regular process of asking them for feedback at midterm and at the end of the class but also just by watching them. If I see some of the students are going to sleep, that encourages me to have them break into groups and do things where they have to talk to each other, think and share, rather than just sitting there listening to me.

I was thinking just this morning as I was reading the Book of Mormon about how many lessons there are in every verse. How much could be done to dig deep into each verse and how profound the experience could be if we could spend 10 years in the Book of Mormon class instead of just 4 months. Also, the fact that there are innumerable resources beyond the Book of Mormon itself that we could use leads me to the conclusion that there is no possible way for me to “cover everything in the Book of Mormon.”

I was talking about ways that my evaluation of my teaching parallels my evaluation in more formal or paid evaluation settings too. In addition to the stakeholder attention that I make toward students and myself, there are all the other issues associated with stakeholder analysis such as the values that I have and values students have. I try and understand what they are, what mine are, what goes together in meaningful ways.

In addition to wanting to match my criteria with those of the student stakeholders, I think we're exploring different kinds of evaluands. I've talked about myself as an evaluand and me wanting the students to be pleased with me and how I'm teaching them. Definitely the Book of Mormon or the field of evaluation or qualitative research becomes an evaluand that I want the stakeholders/ the students to value. The students and their progress are evaluands that I spend lots of work tending to in my teaching. And I think there are probably lots of other evaluations going on there as well.

The key difference in all this is that I'm not thinking in terms of stakeholders, their criteria, or their definitions of the evaluands and so on when I'm doing evaluation as a teacher. Instead I'm just teaching and thinking about those issues without putting those names on them. What difference does that make? Would it make more of a difference in how well I evaluate and how well I talk if I thought in those terms and talked about it in those terms regularly with students? I’m not really sure?

I may come back later and talk about some of evaluations that I make in daily life and see how those match up with the evaluations I have talked about so far.

C. Oct 14- So, now it is later. I’ve spent quite a bit of time today editing the material I dictated using Dragon above. I suppose it was worth doing it that way since I did the dictating while driving to SLC but I think in the future I’ll audio record while I’m having Dragon write so I can figure out the confusing parts later.

Anyway, one example of me evaluating in daily life is my evaluation of Rachael’s family’s visit here this summer. She asked Denise and me for our evaluation of the summer and so I’m going to give it here and then reflect on it a bit.

First, I was surprised Rachael asked for this evaluation. That isn’t something I’m used to. I think she did so because she felt a little uneasy about how it went. I noticed so apologized a few times this summer about how she felt they were taking up too much of our time and energy. I think my main focus during the summer was on evaluating how she was evaluating how it was all going.

Second, there were several things I really liked about their visit and a few things I didn’t like so much. I think that is usually how I evaluate things, experiences, events in my daily life. There is hardly any thing that is 100% good or 100% bad or in need of improvement.

I imagine that identifying what I think/thought was good and not so good will reveal the standard items I look for in all evaluations—how I was viewing multiple stakeholders and their varying ideas about what evaluands are most important and their varying values or criteria for judging those evaluands. How I gather data, interpret it and make the evaluations is all pretty implicit. Right now, it is me sitting here and reflecting back on the summer and trying to remember feelings, thoughts, and experiences/conversations we had.

So, what was good?

I miss all my family when they aren’t around and so any time we get to see them and be around them is cherished. That was the case here. When Rachael said they were thinking of coming for a month or so, I got really excited and hopeful that it would happen.

When it actually did happen, Denise and I drove up into Idaho to meet them and drive back with them so Rachael wouldn’t be too tired. I got to have Owen (he turned 7 later in Nov 2011) in the car with me and Denise rode with Rachael and the rest of the boys. That was fun, to have Owen to myself. He didn’t say much and so I ended up listening to the audio book that I had been listening to before we met them. I felt a little badly about that—like I should have been conversing. But I was also hoping to get to know him as he was and without trying to get him to be a certain way. So, I felt good about that time and hoped he did too. In the past, Owen has been the hardest one for me to get along with. He says things that surprise me and I respond negatively to them at times. So, I was glad we had a little time together (about 3 hours) and we visited a little but not much. He sort of nodded off now and then in the back seat in his car seat and I mostly listened to my book as I drove along.

I probably need to review my journal to remember the details of the rest of the summer and how I evaluated them. But here are some general evaluation feelings I recall-

I was pleased to see the twins (almost 11 years old) playing so well together. That always satisfies my hopes that they will be friends. I was happy to see that Owen and Tommy (4) and George (1) were getting along pretty well with each other too. Before Tommy seemed pretty jealous of George but this time I didn’t notice much of that. The twins and the younger three have always gotten along, as far as I can tell and they did this time too.

I was happy to see that Rachael was comfortable with us enough to bring her five boys without James and spend that much time here. That is an indication to me of a lot of trust and peace (which I’m a little worried might have been compromised because she is asking to hear our official evaluations now).

I was pleased to think that James trusted us enough to let Rachael come and bring the boys without him.

I was happy that Denise got to be around Rachael as much as she did. I know that she values Rachael’s views on things, feels close to her as a daughter, close to the boys as their grandma and wanted to spend this time with them.

Likewise, it was a happy thing for me to see Rachael’s siblings and their children being around her family. I like to see them bonding, being friends, forming memories they’ll cherish, and so on.

And I too had many happy hours just being around the family and seeing them interact with one another and with others. I was glad they were here when I wasn’t busy teaching so I could spend time going to Thanksgiving Point, on walks around the neighborhood and BYU, swimming, to a movie, and elsewhere with them. I didn’t go everywhere that Denise went with them. I guess I felt she needed time with them without me around or something? I also did have to go to physical therapy and a few work events and to work out which interfered with some of the social moments I could have had. I did take the twins to the Monte Bean Museum by myself once and I took George and Ruby on a walk in the stroller once while Denise and Rachael took everyone else up to the Y. I couldn’t really hike the Y with my knee this summer, so that worked out well.

So, those are my general evaluations of things that went well.

Here are some thoughts about how things might have been better.

I found myself being annoyed when the kids (not just Rachael’s) would plead for food, especially right after they had just had some and thrown a bit of it away. I remembered feeling the same way with our own kids and I didn’t like the memory. I feel some self-evaluation in this evaluation because I think I have a lot of issues with food that haven’t been resolved. I grew up believing I had to eat everything I put on my plate and I also didn’t grow up (at least I don’t remember it) eating much between meals. Somehow, I think that because that was the way it was for me and because I don’t like to drop everything and get food ready for people, and I don’t like to see Denise or others drop everything to feed kids who didn’t eat their dinner when it was time. So, I let that bug me more than it should have. Again, the main people in Rachael’s family I found bugging me about this were Owen and Thomas. With George, I just expect babies to be hungry all the time. And with the twins, they just seemed to get and stay full during meal times. But O and T and some of Ryan’s kids pushed my buttons a few times over food.

There were some times when I let this feeling of annoyance track over to Rachael because it seemed to me that she was willing to feed them more often than I would and to let them throw away things when they didn’t like them, more often than I would. But I felt guilty having any of those feelings because I am pretty sure that I’m too extreme in the other direction and I’m also not their parent and she is. I worried that maybe I was too extreme when she was a child in our home and maybe I am getting back to me the results of how I was as her father!? I didn’t want to say anything to her about it for fear of revealing my fears and guilty feelings but also because I want her to feel completely comfortable raising her kids however she and James work that out.

I suppose that is an underlying reason she is asking for this evaluation—she probably felt those feelings even though I didn’t share them. And now what is she going to think when she reads all this? Maybe it is far enough down in the document that she won’t? I have to admit though that when she was complaining to me and Denise on the phone/webcam the other day and showing us how George is constantly begging for food and then just eats one bite from an apple and throws it on the floor, I wondered if she was feeling some of what I feel when kids do that less intensely than George was doing it?

Most of my negative evaluations of the summer are in that previous long paragraph about food. The other area that got to me from time to time was the intensity of the cousins’ play with one another to the exclusion of all adults, especially me. I was really disappointed when the grandkids came to the house and completely ignored me, even when I was doing my best to reach out and be friendly to them. This happened most when Ryan’s kids were here with Rachael’s kids and a little bit with Trent’s kids but they weren’t here that much. I loved seeing them enjoy one another but I felt excluded and sad that they didn’t seem to care if I was there or not.

On the other hand, this isn’t something I want Rachael or Ryan to improve. I would feel terrible if they bribed or pushed their kids to go say hi to Grandpa before they could have fun with their cousins. I just wish I had a better relationship with them so they wanted to include me a bit more. I worry that maybe I’ve burned bridges with them or something. But it is probably just that they enjoy each other so much, right?

Along those lines, Denise spent many evenings scratching backs and reading with the boys and I only joined them in that a couple of times. Why? That would be a way to build a relationship and I probably should have done it more often. But I don’t really like doing it for some reason! Maybe I feel manipulated because it seems to go on and on and Thomas was especially demanding about it rather than being appreciative the few times I did it. I also felt that they had this special bond with Denise that I was infringing on in some ways. Maybe I made that up as an excuse, not sure. Anyway, not being part of the night time routine was a negative in the long run, though I thought of it as a positive during the summer because I could do some other things while Denise was spending quite a lot of time in their bedroom scratching their backs and reading to them.

I had planned to interview Rachael a few times while they were here and we only actually did it twice (and I have lost the recorder I was using when Bonnie Busco came to visit and our interview terminated). I have evaluated that whole experience negatively because I lost that recorder. I had some good stuff on there and I wasn’t taking notes so I have nothing. And I felt like Bonnie’s appearance was kind of symbolic of why I didn’t get more interviews. Rachael is a busy woman and she isn’t just sitting around waiting to talk to me either. But she was able to put aside whatever else was happening to visit with friends who came, including Bonnie. That is good and it is similar to how Denise is—she puts out energy for visitors and then collapses into bed but my only chance to get a good visit is if we can take a walk together. I didn’t find a good way to have a visit with Rachael. I could see that she had very little time and when she did she spent it with James on their iPhones or with her friends who came to visit or with her kids. I got in some good visits at the pool when I went with them but I couldn’t really record an interview there. She offered a few times to let me interview her but I felt like I was imposing and that she had other things she really ought to be doing so I declined. That was a mistake.

So, although I could probably get even more into here by reviewing my journal, here are my thoughts about this personal evaluation I’ve been doing as seen through my evaluation framework—

Stakeholders- I’m definitely the primary one. I evaluate myself negatively on that and believe I should be paying more attention to the other stakeholders. I should be asking more about how Denise, Rachael, and the other family members are feeling about things instead of worrying about how I’m feeling about them. But when it comes to most of my personal evaluations, I do focus a lot of attention on my definitions, my values, my criteria, my standards. Yet, I do see acknowledgment of others and their views in what I’ve written above (maybe because I know others will probably read this and I don’t want to come across as selfish as I probably am?)!

Evaluands- They tend to be the stakeholders too, including me. I was evaluating myself in much of what was written above and also implying that I was evaluating Rachael, her boys, Denise, and our other children and grandchildren. Maybe my relationships with these people are evaluands too? Rachael invited us to evaluate the whole summer and her being here. Kind of a big task but I think most of what I said gets at that too.

Criteria- I have a lot of (probably impractical) shoulds and oughts in my mind and life when it comes to my relationships with my family. I have expectations of them, me, and us together. Many of these are revealed above. I expect kids to be hungry and full when I am! I expect them to be more mature than I am! I expect them to like me without me having to do much of anything to earn that love and respect. I expect myself to be happy with every situation a grandchild helps create. I think my criteria are unrealistic.

How do I gather and interpret and evaluate data- As noted above, I have data in my journal but I’m reticent to go look at it! So, I’ve just written my reflections in here for about an hour. That is my data gathering and analysis process—remembering, thinking, wondering what to write and share with whom, and now I’ll be sending it out to Rachael (and maybe others) to see what they say and think and feel. That will probably influence my feelings about it all too. So, things will keep rolling along and constantly changing as new stakeholders get involved and as relationships among stakeholders evolve and as we each change over time, and so on.

Conclusions

I’m concluding from this 10 pages or so of reflection that our evaluation lives are much more complicated than I had thought. It isn’t just a matter of comparing what is to what should be in some rational methodological way. We are constantly weighing and defining our relationships with others (stakeholders) and using those relationships to reconsider our values, criteria and standards for different versions of the evaluands we are considering. If we’re constantly readjusting all those things, how can we come up with formal evaluation questions and then design formal data gathering and analysis procedures to answer them? We don’t have time for all of that business and don’t believe it would help us that much if we did, at least not in our daily evaluation lives!

Implications

So what are the implications of all this for how I evaluate differently in my daily life versus in my teaching life, versus in my professional evaluator life? Here are some patterns I think I see-

The more daily life evaluations are less formal because there are so many of them and most of them take place inside me without really consulting with others. But the formal ones are not as formal as I talk about them being. Most of my evaluations of students are informal, with a few formal ones coming when I review their work and the points they’ve assigned themselves or when I’m listening to them in class discussions. But those are the occasions we call evaluation or assessment and we make big claims about them, even though they only account for maybe 10% of the evaluations I do as a teacher. And when it comes to formal evaluations I do as a professional evaluator- it is probably the same. The thing I like about Developmental Evaluation is that Patton has made it seem okay to do lots of informal evaluation and to encourage clients to join professional evaluators in doing these together. But we still have to defend what we’re doing in proposals and reports and occasional presentations to clients and stakeholders. So, we use the language of design, method, meta-evaluation criteria and so on, even though most of our evaluations don’t entail thinking about those things much.

Would my many informal evaluations be better if I used the formal language and standards of evaluation more systematically during them? Is that practical? Could I or should I slow down to do that sort of thing? What other differences besides slowing me down would that make? Better quality evaluations? Better recommendations and subsequent actions? Not sure.

Finally, does understanding my informal personal evaluation life allow me and others who are engaging me in more formal evaluations such as when I’m teaching them or learning from them, or when I’m doing professional evaluations for them, support me and them in better ways to transition from the formal evaluations to my informal evaluation life? Can the formal activities take into account the emotional and complex and constantly changing evaluations that I’m making and shape them so I’m more likely to use the results of the formal activities more intelligently?

So, now I’m wondering how Denise and Rachael and others who know me well will respond to all this? I guess the only way to find out is to send it to them and ask! That’s what I’m doing with this. If you get this far, please reply and tell me your thoughts!