Evaluating success to accelerate impact

PATH works with local experts like Medical Officer Tolbert Shaba, to understand how and why public health programs work. Kazangula District, Zambia. Photo: PATH/Gabe Bienczycki.

Understanding why a public health program works (or doesn’t) is a complicated task. In many cases, governments, implementing partners, and donors are all working together to address the same health challenge through a variety of related and complementary projects and programs.

Governments and donors have a vested interest in understanding how these programs work. How do they improve health outcomes? And what barriers are keeping them from even greater health impact?

As new innovations are implemented, governments, donors, and partners need information to adjust in real time, reflect on successes and challenges, and improve design and scale-up of programs.

Implementation research (sometimes called implementation science) considers how our work translates from initial plans and ideas into real world health impact. By looking across the full spectrum of implementation—inputs, activities, outputs, outcomes, impact—using a suite of data sources and methods, implementation research helps us learn what is going well, what we can do better, and how we should continue to evolve.

Over the last seven years, PATH and partners have implemented prospective assessments in Bangladesh, the Democratic Republic of the Congo (DRC), Guatemala, Mozambique, Senegal, Uganda, and Zambia.

Gavi Full Country Evaluations

In partnership with the Institute for Health Metrics and Evaluation (IHME) and working with evaluation partners in Bangladesh, Mozambique, Uganda, and Zambia, we designed prospective evaluations (2013–2018) to support ongoing learning and improvement of Gavi and national immunization programs.

The aim of these evaluations was to identify drivers of vaccine equity and coverage.

The findings and recommendations from the Full Country Evaluations have been used by Gavi and country governments to refine their approach. For example, Gavi revised its policies around the introduction of human papillomavirus (HPV) vaccines to increase learning to inform scale-up.

Global Fund Prospective Country Evaluations

The Global Fund Technical Evaluation Reference Group commissioned a set of independent evaluations to guide the development and implementation of Global Fund investments in eight countries. PATH and its partners are conducting these evaluations in DRC, Guatemala, Senegal, and Uganda (2017–2021).

These evaluations create timely evidence to support program improvements and accelerate progress.

Findings have influenced the work of the Global Fund and its partners. These evaluations provide in-depth knowledge of the complexities of implementation that country stakeholders can use for learning and improvement. Comparing evidence across countries helps identify common implementation bottlenecks for the Global Fund to address through changes to its policies and processes.

PATH provides experience and expertise to conduct rigorous, high-quality evaluations:

• Expertise in evaluation design and methods.
  PATH has deep technical expertise in conducting evaluations—from determining research questions and evaluation design to mixed-methods data collection and analysis.
  
• Focus on strengthening in-country evaluation platforms.
  PATH has strong relationships with both global and in-country evaluation partners to foster country ownership and strengthen capacity for implementation research.
  
• Commitment to learning.
  PATH continually strengthens and refines our approach to respond to changing needs and emphasize continued learning.

So how do we do it? PATH combines a variety of research approaches that generate both quantitative and qualitative evidence. This mixed-methods approach ensures a breadth of evidence and a way to validate our findings.

Although the evaluations for Gavi and the Global Fund were multi-year, multi-country efforts, there is a growing need for fit-for-purpose, rapid evaluations in global health. PATH developed Rapid Evaluation, Action, and Learning Toolkits that allow us to apply implementation research principles in a flexible and rapid way through root cause analysis of barriers to implementation, followed by designing, testing and refining solutions.

PATH is committed to continual learning—including capturing lessons about how we learn. Sharing lessons about our evaluation process is another way to advance high-quality implementation science in public health.

Reflecting on the Gavi and Global Fund evaluations, PATH has shared these lessons:

• Engage stakeholders early.
  Engaging governments, donors, and partners from the beginning leads to stronger evaluation design and greater participation.
  
• Balance objectivity and learning.
  Including research components in ongoing public health projects creates opportunities for data collection and adaptive management—but implementation researchers should also ensure a level of objectivity to make findings more robust.
  
• Design for flexibility.
  Public health is constantly changing, so evaluations should be designed with flexibility to meet changing needs and circumstances.
  
• Build research teams intentionally.
  Team members that bring different skills and experiences—complemented with the right tools and methods—expand evaluation options and strengthens finding and recommendations.
  
• Communicate the strength of evidence.
  In mixed-methods research, some evidence will be more rigorous than others. When translating evidence into recommendations, noting the strength of the evidence is an important guide for decision-makers.

PATH will continue learning, adapting, and evaluating in our efforts to strive for greater health equity.

In addition to PATH, the evaluation teams for the Gavi Full Country Evaluations and the Global Fund Prospective Country Evaluations included a coalition of organizations:

• Institute for Health Metrics and Evaluation (IHME), University of Washington (USA)
• International Centre for Diarrhoeal Disease Research (icddr,b) (Bangladesh)
• Universidade Eduardo Mondlane, Health Alliance International, Manhiça Health Research Centre, and National Institute of Health (Mozambique)
• Infectious Diseases Research Collaboration (IDRC) (Uganda)
• University of Zambia (Zambia)
• Centro de Investigación Epidemiológica en Salud Sexual y Reproductiva (CIESAR) (Guatemala)
• Institut de Santé et Développement/Université Cheikh Anta Diop (ISED/UCAD) (Senegal)

Gavi, the Vaccine Alliance, and the Global Fund to Fight AIDS, Tuberculosis and Malaria provided support for the design and implementation of these evaluations.

• Global Fund Prospective Country Evaluations
• Gavi Full Country Evaluations
• Lessons learned from implementing prospective, multicountry mixed-methods evaluations for Gavi and the Global Fund
• How to evaluate the implementation of complex health programmes in low-income settings
• Partnership and participation—a social network analysis of the 2017 Global Fund application process in the Democratic Republic of the Congo and Uganda