Update on progress

DDW Fieldnotes 9/23/2011—Research and Project Ideas

Recorded 9/23/2011 and transcription finished 9/26/2011

Research Framework.amr File

DDW: Okay, I’m walking along thinking here. I want to talk about my research project ideas. Yesterday I got working on analyzing some of the stories from Terkel’s “Working” book. And they’re so rich and full of evaluation. So I wanted to start sorting out some of what I’ve got.

1. First, I have sources of lived evaluation stories from “working”, from all the other books Studs Terkel has written and that other people like him have done, including most biographies. Which would include ethnographies and  phenomenologies (like the ones by Max VanManen and his students at http://www.phenomenologyonline.com/sources/,

(1min) journal articles, stories, others. There’s an endless stream of those.

2. I also have my own case studies that I’m conducting with people. And one thing I’ve struggled with those is to know how deep I should take them. Should I go and do a full life story for each person or should I just ask each person to tell me, kind of like I did with one lately, one interesting dimension of their evaluation life?

3. And then the third source is the world of fiction, these novels that are rich with evaluation examples that people respond to.

(2min) When I say novels I would include anything from the humanities, like movies, art, short stories, longer novels like Tolstoy’s that I just read this summer and so on. So there are those three big categories of sources that I’m aware of right now, there are probably many others.

So another thought that I’ve been having about that is: Once I have these stories what do I do with them? What I began doing with the “working” stories yesterday was seeing patterns across them. As soon as I started doing that I realized I can see the patterns in many different dimensions: (3min)

1. There are the patterns according to what they, the story teller evaluates,

2. what the criteria are that they use to evaluate.

3. which stakeholders they take into account when they evaluate-- many, many dimensions.

So then I started thinking: Okay maybe selecting the focus of the evaluation should be centered around answering three main “so what?” questions. And the “so what’s?” that I’ve thought of right now are: 1. So what for understanding what humans are? 2. what are the implications for education, for teaching people to be better evaluators? 3. After these first two of just understanding the human experience and looking for learning and teaching issues and dimensions, is “so what for evaluation theory in the field of evaluation?”

1. The one that Greg articulated in the MUR class the other day: So what

(4min) for understanding what humans are? How do humans function, how is evaluation played out as a key role in human being? That in itself would be very interesting and probably ought to be the bulk of any sharing that I do of what I’ve learned from these case studies. I could do that with a single case and just basically lay it out, kind of like Tolstoy does; as a story, but more like Terkel maybe. Maybe elaborating beyond Terkel because his are just short little vignettes. So I could take vignettes (5min) and expand them. I could have some very interesting cases that would be, rich, phenomenological, lived experiences stories, about how these people experience evaluation in their lives.

In doing that I’m thinking that there are at least these categories:

a. their evaluation lives in their work life or careers, which could include career decisions, career training so that’s where you get into the education explicitly, um.

(6min) So that’s the career area.

b. Another area would be home life. Evaluations made as a member of a family. And that could be expanded I suppose to include the community that they are being a family in.

c. A third category could be an evaluation in their personal growth life, which could get into all the dimensions of learning, their psychological health, etc. Here, I’m thinking of this TV series Monk, which brings out not only Monk’s growth and development and choices but those of Randy and Teager, and (7min) Stottlemeyer. So here this overlaps a lot with career. It would be hard to separate career from personal learning but I’m thinking that we do have these different dimensions of life. If I don’t do a full life case study of each person I’m probably going to have to focus in on that part. But maybe using the holograph or hologram idea, we could say that each case study could include just whatever it is their lived experience is and it will tap into all these categories.

(8min)

d. Another category might be religious life or spiritual life. Um, how are people and articulate that. And there are probably many others but that’s the beginning. So this all in the first main purpose of trying to understand human experience evaluatively.

2. A second “so what” question is what are the implications for education, for teaching people to be better evaluators? One implication might be we don’t have much to teach because people are already excellent evaluators. Another one might be they are really excellent in some areas but there are other areas we’re all weak in and education could do something about that. (9min) This is kind of going along the lines of the talk that Michael Scriven gave in 1999. It was his presidential address for the American Evaluation Association in which he said that we as evaluation professionals have a responsibility to help people in all dimensions of their lives to be better evaluators. And when I heard that and as I thought about it in the twelve years since I thought: yes, in schools we’re trying to teach people to evaluate different disciplines, to evaluate their knowledge of disciplines and to evaluate using the standards and the means of those disciplines (10min) and that’s what I think Scriven meant when he said that evaluation is a trans discipline. So there is that dimension of what can the regular education system do to help people be better evaluators.

Perhaps this ties into the critical thinking movement. Critical thinking I think is promoting a certain kind of evaluation. But again each discipline is also doing that. We evaluate things mathematically, scientifically, through the eyes of the humanities, which involves critical analysis from many different dimensions. So there’s all that going on and can we learn anything from these case studies to do all that better? (11min) And can we come up with courses in Universities and K-12 settings that would allow us to invite people to think about their evaluation lives and to work to improve them. I could imagine commercial courses that would help people with this because there’s so many such things-- self help books and courses that are meant to help people get outside their boxes they sometimes create for themselves. And I see this over and over in psychogy (12min) studies and so on.

Basically people are trying to help others have a different evaluation paradigm. That’s one way of thinking about that. So I’d like to explore: what are the learning, education, training issues that might be relevant to improving people’s evaluation lives or helping them improve their own evaluation lives just by thinking and reflecting on them. Keith has told me that just talking to me about how he’s evaluating his approach to his master’s thesis, just talking about that has really helped his refine his evaluations about (13min) how people evaluate themselves when they have learning failures. That whole area of recognizing learning failures, thinking about evaluation as a form of learning and exploring how to improve that learning by improving how we evaluate seems like a rich, productive possible area.

3. Finally the third “so what” area, after these first two of just understanding the human experience and looking for learning and teaching issues and dimensions, is “so what for evaluation theory in a field of evaluation?” (14min) That’s the one that I got started on all this with when I read Schwandt and heard his argument that evaluation theory has been so consumed with trying to tell people what to do from a social science perspective and from other perspectives as well. If you look at Alkin, he has his theory tree and shows three different main trunks or roots that he’s analyzed evaluation theory into. But none of those, as far as I can tell, are based on really understanding what people do. And none of them are based on the humanities, which is what (15min) Schwandt was inviting us to look to the humanities to see what they could tell us about evaluation that could help us as we develop evaluation theory.  

So, my question would be, using more of an inductive or maybe abductive method, which I’ve been learning about, I’m going from the cases that I’m describing to the evaluation theory. Maybe there would be some implications for how we do evaluation formally. I would like to look in particular at how some of the cases might be involving people being involved in a  (16min) formal evaluation and merging that experience with all the other evaluations that they are doing.  I’m sure that will come up naturally.  And I would like to explore that intersection between a given person’s ongoing evaluation life and all of the dimensions that I’ve talked about and how it is shaped and perhaps modified, expanded, contracted and so on, when they encounter a formal evaluation.  

So, for example maybe I could use a faculty interview as an example. (17min) I was talking to a faculty member in his interview this week and he talked all about how he judges success of his efforts to invite people into the topic he studies so much.  In the middle of talking about that, he said a couple of times that his wife is very supportive and how important it is and that they’re actually a team.  He maybe didn’t use that word but something like that.  And when I hear that I think there’s an overlap here between the evaluation he’s saying about himself and his career, and the evaluation he’s doing of himself as a husband  (18min) and a father and all the other dimensions that mentioning of his wife brought up.  So then I also know that he’s very involved around now having his tenure reviews and thinking about going up for advancement.  Both of those involve formal evaluation by the University of him and how well he’s doing, not only in his research area but in all dimensions of his work. (19min) It’ll be interesting to explore with him how those worlds intersect.  

Or, maybe they are so intersected, so intertwined, that he would have a hard time choosing what goes out. But would teasing them out help or attempting to be helpful in terms of talking about the world of personnel, evaluation, and the personnel evaluation standards, and what the formal field of evaluation has been saying about not only personnel evaluation but self evaluation and how self evaluation intersects with external evaluation by supervisors, by colleagues and so on (20min) and university teaching, and scholarship and service.  So that’s one example.  

Another one might be… say I’m interviewing a person about their career, say one of these instructional designers that we’ve interviewed from the study with Yancher.  And in the midst of that interview, some of a person’s personal life evaluation comes up and intersects with their understanding of evaluation theory on how to evaluate educational products. (21min) That’s the intersection that I would think would be really interesting to explore so that we can see what implications there are for evaluation theory about evaluating educational products, and then their production, their development, and their design.  But also, what’s going in the lives of the designers and the people they are designing for when a formal evaluation is imposed, or invited, or suggested.  Seems like there was at least one of those designers that was interviewed that said that he had all kinds of things going on in his personal view about how people learn and teach and so on that he wanted to explore (22min) but he had to be kind of forced into a model that the place he worked for used that didn’t allow for that kind of exploration.

The same could be said for other fields: program evaluation involves people.  There are lots of program theories, one of Patton’s big emphases in his theory of Developmental Evaluation is that you look around for people who are likely to use the information that you produce.  Well, that very focus indicates why he’s developed an evaluation theory that is utilization focused and now developmental.  And he doesn’t try and change (23min) all the people and their personal evaluation approaches; instead he looks for people who already have personal evaluation strategies and looks for how he can build off of those to enhance their evaluation use.  So that would be another one that would be interesting to explore.  

Kind of in summary, I’m thinking that this little walk that I just took in this 25 minutes or so of overview is what I’m going to want to use to organize not only my presentation for AEA, (24min) but also I want to embody this somehow into a draft article that I would submit to the two main journals for AEA, and just ask them: are these topics of interest to the field?  Would you be willing to consider articles written in any of these categories I’ve outlined, or in all of them? If they are positive at all then I would think I would need to start figuring out a little framework that summarizes all of that and then isolates which part of that this particular article is going to focus on. And then I could start inserting cases (25min) into different articles using this overall framework.

I just got an email from Rick West saying that he’s got a bunch of data now for me to look at from the creativity study that he’s doing.  And I’m building a table showing all the different cases that I’m working on and what their current status is.  Three of them are being transcribed, at least three interviews are being transcribed.  I’m finding that that’s going really slowly, so I can’t really depend on quick transcription, but my experience yesterday working through some “working” cases that I’ve read, there’s only about five of them, shows that I’ve got much more data than I need (26min) to do the writing that I want to do.  

I basically need to think in my mind that I’m building a library of cases that can be analyzed, interpreted and explored in all these different ways that I’ve just described.  If I can do that then I won’t get so anxious about getting anyone’s complete story or about getting the work done, it’s never going to be done.  Building a library never ends and people’s individual cases, are infinitely long, I believe; because these are infinite beings.  They have existed forever, (27min) they are going to exist forever, and even if I narrow it down to what’s going on in this life, there’s so much there, too much there to do even one case study, say on my grandmother, or my parents, or my children, or myself, or my wife.  It’s like Denise is finding the stories that she’s writing about her mother.  So given that, I just need to relax and enjoy what I am doing, and continue to build the library.  And draw upon it regularly to actually publish something.  I think this framework I just described should help me do that.  Signing off.