Evaluation Masterclass

Advanced Module, Newspeak House 2024-25

Andreas Varotsis

Introduction

  • Importance of Evaluation:
    • Demonstrates impact, accountability, and guides improvements.
  • Treatment Effect:
    • Measures causal impact of interventions.

Why Evaluation Matters

  • Crucial for funding and credibility.
  • Essential for learning and continuous improvement.
  • Helps organizations focus resources effectively.

Treatment Effects: Why They Matter

  • Shows causal impact, not just correlation.
  • Essential for effective decision-making and resource allocation.

Challenges: Correlation vs Causation

  • Correlation ≠ causation.
  • Importance of a counterfactual (control group).

Example: - Youth employment improvements might coincide with economic growth, not just program impact.

Rigorous Methods: Randomized Controlled Trials (RCTs)

  • Gold standard for impact evaluation.
  • Participants randomly assigned (treatment/control).

UK Example: - Family Nurse Partnership Scotland Evaluation

Rigorous Alternatives: Quasi-Experimental

  • When RCTs aren’t feasible.
  • Propensity score matching, difference-in-differences, regression discontinuity.

Case Study: Family Nurse Partnership (FNP), Scotland, using natural timing differences as controls.

Case Study 1: Read Easy UK

  • Volunteer-led adult literacy improvement.
  • Personal narratives (e.g., Jay Blades).

“Over 200 adults gained literacy skills through personalized coaching, enabling life-changing improvements.”

Read Easy UK

Case Study: Code Club

  • Free coding clubs for children aged 9-13.
  • Over 90% improved in confidence and problem-solving.

Narrative: “Code Club transforms children’s lives by boosting digital skills and creativity, reaching over 200,000 young people globally.”

Code Club Impact

Case Study: Skills for Life Strategy

  • Massive UK adult literacy and numeracy initiative.
  • Despite extensive investment, numeracy levels declined.

Lesson: Outputs don’t guarantee outcomes. Importance of targeting and meaningful metrics.

Skills for Life Evaluation

Case Study: Sure Start

  • Integrated child services for disadvantaged families.
  • Initial evaluations found minimal impact; long-term studies showed significant positive outcomes.

Narrative: Early struggles can lead to later successes; patience and continued evaluation essential.

Sure Start Impact Study

Famous Failure: D.A.R.E.

  • Popular anti-drug education program.
  • Rigorous evaluations showed no effect on drug use.

Lesson: Popular ≠ effective. Importance of evidence-based design.

D.A.R.E. Evaluation

Qualitative vs Quantitative

  • Quantitative: Numbers, measurable outcomes.
  • Qualitative: Stories, feedback, context.

Best Practice: Use both to tell a comprehensive story.

Selecting Metrics

  • Outcomes vs outputs.
  • SMART criteria (Specific, Measurable, Achievable, Relevant, Time-bound).

Example of Good Metrics: - Percentage increase in civic participation. - Long-term housing retention for homeless interventions.

Summary & Key Takeaways

  • Evaluation demonstrates impact and drives improvement.
  • Treatment effects require rigorous methods.
  • Effective communication combines data and storytelling.
  • Avoid common pitfalls with clear planning and meaningful metrics.

Q&A and Discussion

  • Your experiences?
  • How could these concepts apply to your projects?