Criterial and Interpretive Evaluation

class: center, middle, inverse, title-slide

.title[
# Criterial and Interpretive Evaluation
]
.subtitle[
## Keys to Merit, Worth, and Significance
]
.author[
### Dr. Abhik Roy
]

---

<style>

section {
    display: flex;
    display: -webkit-flex;
}

section p {
    margin: auto;
}

section {
    height: 600px;
    width: 60%;
    margin: auto;
    border-radius: 20px;
    background-color: #212121;
}

section p {
    text-align: center;
    font-size: 30px;
    background-color: #212121;
    border-radius: 20px;
    font-family: Roboto Condensed;
    font-style: bold;
    padding: 15px;
    color: #bff4ee;
}

.center2 {
  margin: 0;
  position: absolute;
  top: 50%;
  left: 50%;
  -ms-transform: translate(-50%, -50%);
  transform: translate(-50%, -50%);
}

</style>

<div>
<style type="text/css">.xaringan-extra-logo {
width: 110px;
height: 128px;
z-index: 0;
background-image: url(/Users/skynet/Documents/WVU/Teaching/GitHub.nosync/edp611/static/img/course_hex.png);
background-size: contain;
background-repeat: no-repeat;
position: absolute;
top:1em;right:1em;
}
</style>
<script>(function () {
  let tries = 0
  function addLogo () {
    if (typeof slideshow === 'undefined') {
      tries += 1
      if (tries < 10) {
        setTimeout(addLogo, 100)
      }
    } else {
      document.querySelectorAll('.remark-slide-content:not(.title-slide):not(.inverse):not(.hide_logo)')
        .forEach(function (slide) {
          const logo = document.createElement('a')
          logo.classList = 'xaringan-extra-logo'
          logo.href = 'https://edp611.asocialdatascientist.com'
          slide.appendChild(logo)
        })
    }
  }
  document.addEventListener('DOMContentLoaded', addLogo)
})()</script>
</div>

# Welcome!

Some information about the audio files. Firstly and importantly, I'm no Morgan Freeman but each page will have some audio that will hopefully help you understand a bit more about what's on the page

<center>
<audio controls preload="auto">
  <source src="audio/criteria/S1_Introduction.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
<audio>
</center>

---

# Defining Evaluations

.bg-washed.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt5[
**Evaluation** is the act or process of determining the ***merit***, ***worth***, or ***significance*** of something or the product of that process.
]

<br>
<center>
<audio controls preload="auto">
  <source src="audio/criteria/S2_Evaluation_Definition.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
</audio>
</center>

---

# Explaining it to Everyone Else

<center>
.bg-washed.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt5[
**Evaluation** answers ***if it works?*** and ***why does it work?***
]
</center>
<br>
<br>

We typically leave the ***if it works?*** to people with grounded knowledge in a particular area - aka *content experts*.

---

# Merit, Worth & Significance

First consider these questions
+ What do ***merit***, ***worth***,and ***significance*** mean to you both separately and combined?

+ Can you think of real world examples of each?
<br>

Now listen to these

<center>
<audio controls preload="auto">
  <source src="audio/criteria/S3a_Merit_Worth_Significance.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
</audio>
</center>
<br>
<center>
<audio controls preload="auto">
  <source src="audio/criteria/S3c_Merit.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
</audio>
</center>
<br>
<center>
<audio controls preload="auto">
  <source src="audio/criteria/S3d_Worth.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
</audio>
</center>
<br>
<center>
<audio controls preload="auto">
  <source src="audio/criteria/S3e_Significance.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
</audio>
</center>

---

# Evaluation

+ Evaluation is the act or process of determining the *merit* (***quality***), *worth* (***value***), or *significance* (***importance***) of something or the product of that process.

<center>
<audio controls preload="auto">
  <source src="audio/criteria/S4_Evaluation_Definition_Extended.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
</audio>
</center>

---
# Evalu…

- *Evaluand*. This is not always as obvious to define in practice as it may seem. Where are the boundaries of a particular program? Do the evaluator, their employer, and various other stakeholders agree on those boundaries?

- *Evaluator*. Who you are matters… more on this later

<center>
<audio controls preload="auto">
  <source src="audio/criteria/S5_Evaluand_and_Evaluator.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
</audio>
</center>

---

# Criteria, Standard, and Indicator

First consider these questions

+ In addition to defining these for ourselves, what are examples from everyday?

+ Which of the three is harder to identify? Why?

Now listen to these

<center>
<audio controls preload="auto">
  <source src="audio/criteria/S6b_Criteria.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
</audio>
</center>
<br>
<center>
<audio controls preload="auto">
  <source src="audio/criteria/S6c_Indicator.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
</audio>
</center>
<br>
<center>
<audio controls preload="auto">
  <source src="audio/criteria/S6d_Standard.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
</audio>
</center>

---

# Purposes of Evaluation - Primary

+ *Formative* - Conducted with the intent to improve

+ *Summative* - Conducted with the intent to inform decision making and/or determine judgment

</audio>
</center>
<br>
<center>
<audio controls preload="auto">
  <source src="audio/criteria/S7_Formative_Summative.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
</audio>
</center>

---

# Purposes of Evaluation – Secondary

+ *Developmental* - To help develop an intervention or program

+ *Accountability* - To hold accountable

+ *Monitoring* - To assess implementation and gauge progress toward a desired end

+ *Knowledge Generation* - To generate knowledge about general patterns of effectiveness

+ *Ascriptive* - Merely for the sake of knowing

---

# Developmental vs Formative Evaluation

---

# A bit more on Developmental Evaluation

</audio>
</center>
<br>
<center>
<audio controls preload="auto">
  <source src="audio/criteria/S10_Developmental.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
</audio>
</center>

---

# Try This at Home

</audio>
</center>
<br>
<center>
<audio controls preload="auto">
  <source src="audio/criteria/S11_Activity.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
</audio>
</center>
<br>
<table class="table" style="font-size: 18px; margin-left: auto; margin-right: auto;">
 <thead>
  <tr>
   <th style="text-align:left;color: white !important;background-color: #212121 !important;">   </th>
   <th style="text-align:left;color: white !important;background-color: #212121 !important;"> What might the purpose of the evaluation be? </th>
   <th style="text-align:left;color: white !important;background-color: #212121 !important;"> What questions might the evaluation try to answer? </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> ***Formative Evaluation*** </td>
   <td style="text-align:left;">  </td>
   <td style="text-align:left;">  </td>
  </tr>
  <tr>
   <td style="text-align:left;color: white !important;background-color: #212121 !important;"> The program has been designed and piloted with a group of 20 principals. What kind of evaluative feedback should the ASDC seek? </td>
   <td style="text-align:left;color: white !important;background-color: #212121 !important;">  </td>
   <td style="text-align:left;color: white !important;background-color: #212121 !important;">  </td>
  </tr>
  <tr>
   <td style="text-align:left;"> ***Summative Evaluation*** </td>
   <td style="text-align:left;">  </td>
   <td style="text-align:left;">  </td>
  </tr>
  <tr>
   <td style="text-align:left;color: white !important;background-color: #212121 !important;"> The program has been delivered to all principals in the county. What kind of evaluative feedback should the ASDC seek? </td>
   <td style="text-align:left;color: white !important;background-color: #212121 !important;">  </td>
   <td style="text-align:left;color: white !important;background-color: #212121 !important;">  </td>
  </tr>
</tbody>
</table>

---

# Introduction to Roles and Styles

- Evaluator Roles: Independent, Topical, External, Internal

- The one that you fill in your evaluation will deternine a lot!

</audio>
</center>
<br>
<center>
<audio controls preload="auto">
  <source src="audio/criteria/S12b_Independent.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
</audio>
</center>
<br>
<center>
<audio controls preload="auto">
  <source src="audio/criteria/S12c_External.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
</audio>
</center>
<br>
<center>
<audio controls preload="auto">
  <source src="audio/criteria/S12d_Topical.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
</audio>
</center>
<br>
<center>
<audio controls preload="auto">
  <source src="audio/criteria/S12e_Internal.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
</audio>
</center>

---

# Defining Culture

> What is culture?

> How do we recognize it?

<br>
<i>Culture can be defined as the shared experiences of people, including their languages, values, customs, beliefs, and more. It also includes worldviews, ways of knowing, and ways of communicating. Culturally significant factors encompass, but are not limited to, race/ethnicity, religion, social class, language, disability, sexual orientation, age, and gender.</i><br>
.pull-right[
-- American Evaluation Association (2011)
]

</audio>
</center>
<br>
<center>
<audio controls preload="auto">
  <source src="audio/criteria/S13_Culture.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
</audio>
</center>

---

# The Iceberg

.pull-left[
<center>
<img src="img/iceberg.png" height="500px" style="background-color:#212121;"/>
</center>
]

.pull-right[
<center>
<audio controls preload="auto">
  <source src="audio/criteria/S14_Iceberg.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
</audio>
</center>
]

---

# Markus & Kitayama (1991)

.pull-left[
<center>
<img src="img/eastvwest.png" height="480px" style="background-color:#212121;"/>
</center>
]

.pull-right[
<center>
<audio controls preload="auto">
  <source src="audio/criteria/S15_Self.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
</audio>
<br>
<br>
<br>
<br>
*Download the [article](docs/%4D%61%72%6B%75%73%20%26%20%4B%69%74%61%79%61%6D%61%20%28%31%39%39%31%29%2E%70%64%66)*
]
</center>
]

---

# Cultural Influences Activity

How might each of the following influence an evaluation’s design and implementation?

- Language of participants
 - Role of food and eating
 - Evaluators attire when interacting with participants
 - Concept of time
 - Communication styles
 - Importance of family relationships
 - Values/norms re: individualism, independence, conflict
 - Understanding of social structure

</audio>
</center>
<br>
<center>
<audio controls preload="auto">
  <source src="audio/criteria/S16_Cultural_Activity.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
</audio>
</center>

---

# Increasing your cultural sensitivity

Try to
- Assess your own attitudes, beliefs, values
- Incorporate culturally diverse groups and perspectives throughout evaluation process
- Build in sufficient time to negotiate access and build trust in community / organization
- Demystify evaluation, avoid jargon, build buy-in
- Be flexible in choice of evaluation methods, use multiple data sources, multiple data collection methods, multiple perspectives

</audio>
</center>
<br>
<center>
<audio controls preload="auto">
  <source src="audio/criteria/S17_Sensitivity.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
</audio>
</center>

---

# Meta-evaluation

- A ***metaevaluation*** is an evaluation of an evaluation or evaluations.

- There are two main types
  - *Formal* - Extra, external study to authenticate the process or product of the evaluation.

- *Informal* - Evaluating your own evaluation.

</audio>
</center>
<br>
<center>
<audio controls preload="auto">
  <source src="audio/criteria/S18_Meta_Evaluation.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
</audio>
</center>

---

# Things to Evaluate

>- Evaluation criteria

>- Evaluation standards

>- Performance

>- Program costs

>- Program goals

<br>
<center>
<audio controls preload="auto">
  <source src="audio/standards/S1 Things to Evaluate.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
<audio>
</center>

---

# Things to Consider

>- Everything on the [JCSEE](https://evaluationstandards.org/program/)!

>- Location

>- Stakeholder Needs

>- Sponsor Needs

>- Your costs

<br>
<center>
<audio controls preload="auto">
  <source src="audio/standards/S2 Things to Consider.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
<audio>
</center>

---

# Goals Include

>- Preparatory conditions
  + What does it take to be ready to target a goal? 
  + Progress toward that state is a goal.

>- Intended operations
  + What a program tries to implement to achieve a goal is a type of goal.

>- Targeted outcomes
  + Short-term (initial)
  + Medium-term (intermediate)
  + Long-term (distant)

<br>
<center>
<audio controls preload="auto">
  <source src="audio/standards/S3 Goals.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
<audio>
</center>

---

# Whose Goals are Important?

It's not so clear cut!

<br>
<center>
<audio controls preload="auto">
  <source src="audio/standards/S4 Whose Goals.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
<audio>
</center>

---

# How to Decide Which Goals Matter Most?

>- Talk to stakeholders/sponsors; explicitly ask their perspective on recipient needs, program goals, and evaluation goals.

>- Figure out other ways to find needs.

>- Read all available program documentation.

>- Develop a logic model connecting needs through program activities, outputs and outcomes associated with goals.

<br>
<center>
<audio controls preload="auto">
  <source src="audio/standards/S5 Deciding on Goals.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
<audio>
</center>

---

# Logic Model Parts

+ **Inputs**. what resources go into a program (aka investments)

+ **Outputs**. what is produced through those activities
  + **Activities**. what activities the program assumes (what actually occurs)
  + **Participation**. who is part of the program and how are they served (who is served like stakeholders and sponsors and what engagement they have)

+ **Outcomes**. changes or benefits that result from the program
  + **Short**. Learning (awareness, knowledge, attitudes, skills, opinions, aspirations, motivations)
  + **Medium** *or* **Intermediate**. Action (behavior, practice, decisions, policies)
  + **Long**. Consequences (civic, economic, environmental, political, professional, social, etc.)

<br>
<center>
<audio controls preload="auto">
  <source src="audio/standards/S6 Logic Model Parts.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
<audio>
</center>

---

## Basic Logic Model

.pull-left[
<center>
<img src="img/Blank Logic Model_transparent.png" style="background-color:#212121;"/>
</center>
]

.pull-right[
<center>
<audio controls preload="auto">
  <source src="audio/standards/S7 Blank Logic Model.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
<audio>
]

<center>
.pull-left[
<br>
<br>
*Download a [high quality PDF](docs/Blank Logic Model.pdf)*
]
</center>

<center>
<br>
<br>
.pull-right[
*More [types of logic models](https://fyi.extension.wisc.edu/programdevelopment/logic-models/)!*
]
</center>
---

## Example Logic Model

.pull-left[
<center>
<img src="img/Example Logic Model_transparent.png" style="background-color:#212121;"/>
</center>
]

.pull-right[
<center>
<audio controls preload="auto">
  <source src="audio/standards/S8 Example Logic Model.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
<audio>
</center>
]

<center>
.pull-left[
<br>
<br>
*Download a [high quality PDF](docs/Example Logic Model.pdf)*
]
</center>

<center>
<br>
<br>
.pull-right[
*More [examples of logic models](https://fyi.extension.wisc.edu/programdevelopment/logic-models/bibliography/)!*
]
</center>

---

## Identifying Criteria

What matters enough to measure given what you know about all the other factors? Especially,

+ Any relevant past performance data

+ Costs

+ Existing literature and standards

+ Needs and Goals

<br>
<center>
<audio controls preload="auto">
  <source src="audio/standards/S9 Identifying Criteria.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
<audio>
</center>

---

## Estimating Cost

+ Often very hard to do well

+ Almost always addressed in a relative way
  
  + Relative resource intensity of alternative programs (including development and/or retraining costs)
  + Proportion of resources allocated to different program activities (logic modeling can be very helpful for thinking about this)

<br>
<center>
<audio controls preload="auto">
  <source src="audio/standards/S10 Cost Analysis.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
<audio>
</center>

<br>

---

## Measuring Performances

>- Sponsor perception

>- Staff performance / perception

>- Stakeholder performance / perception

<br>
In a nutshell

+ Who is best positioned to really know???

+ Multiple times, multiple perspectives

<br>
<center>
<audio controls preload="auto">
  <source src="audio/standards/S11 Measuring Performance.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
<audio>
</center>

---

### Develop an Evaluation Purpose Statement

*Step 1*. Critique sample evaluation purpose statements (see following slides) to better understand what might go into crafting a good one.

*Step 2*. Write preliminary evaluation statement for the program you'll evaluate.

*Step 3*. Write 3 - 7 potential evaluation questions relevant to the evaluation purpose statement you wrote for step 2.

<br>
<center>
<audio controls preload="auto">
  <source src="audio/standards/S12 Evaluation Purpose Activity.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
<audio>
</center>

---

## Thinking About Evaluation Statements

In the next two slides, take a look at the sample evaluation purpose statements and *for each* try to identify:

+ What is being evaluated?

+ How will results be used?

+ What kind of evaluation (developmental, formative, or summative)?

+ How could the purpose statement be improved?

<br>
<center>
<audio controls preload="auto">
  <source src="audio/standards/S13 Evaluation Purpose Questions.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
<audio>
</center>

---

## Sample Evaluation Statements (1/2)

---

## Sample Evaluation Statements (2/2)

---

# Tips

+ Be clear

+ Be concise

+ Be truthful

+ Get buy in

+ Be confident but **never** demeaning or overpowering

+ Don’t overpromise and underdeliver!

+ Write in a way that's meaningful to your sponsor(s)/stakeholder(s)

<br>
<center>
<audio controls preload="auto">
  <source src="audio/standards/S14 Tips.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
<audio>
</center>

---

# Final Thoughts?

What made sense or connected well for you in the things we focus on this week?
What didn’t quite fit together or make sense in your mind?
Anything in the reading you wondered about that we didn’t address?
Any remaining questions about the course, syllabus, assignments, etc.?

</audio>
</center>
<br>
<center>
<audio controls preload="auto">
  <source src="audio/criteria/S19_Final.mp3" type="audio/mpeg">
  Your browser does not support embedded audio.
</audio>
</center>

<center>
<a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br /><br />This work is licensed under a <br /><a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License</a>
</center>