Conference Agenda

Overview and details of the sessions of this conference. Please select a date or location to show only sessions at that day or location. Please select a single session for detailed view (with abstracts and downloads if available).

Please note that all times are shown in the time zone of the conference. The current conference time is: 25th Sept 2022, 09:54:07pm CEST

 
 
Session Overview
Session
ID 738: Rubrics As Tools To Drive Systemwide Change: How To Design An Evaluation And Learning System That Measures What Matters For Equity And Sustainability.
Time:
Wednesday, 08/June/2022:
9:00am - 10:30am

Session Chair: Dr. Thomaz Kauark Chianca, Managing Partner, COMEA Relevant Evaluations
Location: Room - Pjerrot

Session Themes:
Theme 4: Methodological shift: transforming methodologies

Show help for 'Increase or decrease the abstract text size'
Presentations
ID: 738
Tools sessions (tools in use, changing tools) (Abstract Submission 2022)
Themes: Theme 3: Content shift: : transformation in and by evaluation
Keywords: Rubrics methodology, evaluation tools, systems evaluation, rubrics-enhanced evaluation, developmental evaluation, evaluation and learning systems, mixed methods evaluation, systems change, measuring equity and sustainability

Rubrics As Tools To Drive Systemwide Change: How To Design An Evaluation And Learning System That Measures What Matters For Equity And Sustainability.

Presenter(s): Lee A. Risby (Director of Effective Philanthropy at Laudes Foundation), João Martinho (Director of Evaluation and Learning at The Audacious Project, TED), E. Jane Davidson (President at Real Evaluation Ltd.), Thomaz K. Chianca (Managing Partner, COMEA Relevant Evaluations), Katrina L. Bledsoe (Principal Associate at Abt Associates), Michael Quinn Patton (Founder and Director of Utilization-Focused Evaluation)

Laudes Foundation, a family foundation based in Europe, is responding to the dual crises of inequality and climate breakdown through synergistic and collaborative efforts in finance and capital markets, the fashion industry, and the built environment. With an ambition this broad and a global system so huge and complex, what kind of evaluation and learning system could most usefully help guide and evaluate the change?

Key Performance Indicators (KPIs) had given focus and direction to the foundation staff and trust and confidence to Board members, who were used to assessing (and viewing) progress using KPIs (sales, revenues, costs) measured quarterly against targets. Nonetheless, these were not systemic nor focused on quality, and they often diverted the foundation from addressing root causes in pursuit of achieving targets.

In this lively session, we showcase how Laudes Foundation understood the limitations of KPIs and went searching for a better way – a systems change-savvy, rubric-enhanced approach that measures what really matters.

In 2018, the Effective Philanthropy Team proposed that the foundation explore using rubrics to capture mixed streams of evidence for the systemic shifts the foundation wanted to achieve in the fashion industry. An external Overall Effectiveness Evaluation (in 2019) endorsed the shift away from KPIs to rubrics, as well as increasing use of developmental evaluation.

In 2020, this work was expanded as Laudes defined its strategy and systemwide theory of change (https://www.laudesfoundation.org/how-we-work). These were two essential ingredients for the development of the rubric-based measurement and learning system. The approach is grounded in a systemwide theory of change from which rubrics were developed to track key outcomes and impacts. Rubrics allowed for a more nuanced and systems change-grounded approach, one that could flex as change emerged (https://www.laudesfoundation.org/grants/rubrics).

Key principles and approaches in developing these rubrics and the evaluation and learning system that sits around them included: (1) Combined bottom-up and top-down intensive stakeholder involvement; (2) Keeping the evaluative eye on the big picture (not getting lost in details); (3) Embracing complex adaptive systems concepts (tipping points, building on momentum, “good enough” strategy); and (4) A “minimum specs” measurement system based on rubrics (clear shared language across the system, flexibility to adapt to different contexts, to support emergent change across the entire system).

Designing the evaluation and learning system was just one part of the story. Another was how to get buy-in to something that was a significant departure from the previous way of doing measurement based on KPIs. What did it take to get the necessary buy-in? How is Laudes building the new capabilities needed? What else has been key to a successful rollout? We will hear from the leaders and influencers within the foundation whose task it was to persuade their senior colleagues.

Our two eminent discussants will enrichen the session. Michael Quinn Patton will comment on how this approach helps address high-stakes global issues with complex, emergent, systemic change. Katrina Bledsoe will share her reflections about the cultural responsiveness of this theory-driven evaluation system and how well it addresses equity concerns.

Biographies
Lee Alexander Risby, PhD.: Director of Effective Philanthropy at Laudes Foundation. He has over 20 years of experience in evaluation, organisational learning and capacity development in international development, corporate and private philanthropy, and previously held positions at C&A Foundation, African Development Bank and the Global Environment Facility.

João Martinho: Director of Evaluation and Learning at The Audacious Project, TED. He has over 10 years of experience in the philanthropic, public and private sectors, in co-creating and building capacity for effective use of MEL. Previously held positions at Laudes Foundation, at a Brazilian foundation and at a consultancy firm

E. Jane Davidson, PhD.: Best known for developing evaluation rubrics as a methodology for drawing conclusions about quality and value. Jane was the 2019 recipient of AEA’s prestigious Paul F. Lazarsfeld Evaluation Theory Award and is sought after internationally for her signature approach of methodologically robust but refreshingly practical evaluation.

Thomaz K. Chianca, Ph.D.: International evaluation consultant with 25 years of experience leading or managing more than 40 evaluations in 25 countries, dealing with diverse issues – from sustainable and equitable fashion industry to child sponsorship and rural poverty reduction. Thomaz is committed to facilitate evaluations that are relevant, useful, and transformational.

Katrina L. Bledsoe, Ph.D.: An international leader in cultural contexts, cultural responsiveness, and theory-driven evaluation in community-based settings. She specialises in quasi-experimental and mixed-method designs. Her expertise is in community-based education, health and mental health and social services, applied social psychology and cultural responsiveness, and equity in cultural contexts.

Michael Quinn Patton, Ph.D. : Former President of the American Evaluation Association and author of eight major evaluation books including a 5th edition of Utilization-Focused Evaluation (2022) and Blue Marble Evaluation (2020). He regularly conducts training for The Evaluators’ Institute, the International Program for Development Evaluation Training, and Claremont Evaluation Center.


 
Contact and Legal Notice · Contact Address:
Privacy Statement · Conference: EES 2022
Conference Software - ConfTool Pro 2.6.145+TC
© 2001–2022 by Dr. H. Weinreich, Hamburg, Germany