Background: This report is a summary of the impact evaluation that was conducted with the goal of analyzing improvements in health facility supportive supervision (SS) practices after the adoption of the AfyaSS system, and use of the information in the system. AfyaSS system is a digital system that was developed through the collaboration between the Ministry of Health (MOH), president's office —regional administration and local government (PO-RALG), PATH and other stakeholders. The AfyaSS system aims to improve the management and use of health facility supervision data for better health service delivery in Tanzania. It also supports continuity in tracking performance and ensures that better quality data is available and used in managing supervisions.

Objective: The objective of conducting this impact evaluation was to assess the improvement in supportive supervision practices after the implementation of the AfyaSS system.

Method: A multi-case study approach was adopted where both qualitative and quantitative data were collected. The purpose was to get the breadth and depth of findings from a wide range of respondents. A purposive sampling method was used in recruiting participants for the assessment. The main data collection methods used to gather findings of the qualitative approach were key informant interviews (KIIs), observation, artifact analysis, and document review, while data collection for the quantitative approach consisted of online questionnaires. The baseline assessment was conducted from June 6 through August 10, 2021, where a total of 88 interviews were conducted, and 501 online survey responses were gathered. For endline, data collection was conducted from August 1 – 17, 2022 where a total of 100 KIIs were conducted, and 620 survey responses were gathered.

Findings: Generally, the assessment indicated improved data use through real time access of previous supervision reports and plans. Both baseline and endline assessments showed participation of the entire team of supervisors and better coordination was noted as well, when organizing supportive supervision using AfyaSS system. Furthermore, the AfyaSS system had promoted accountability as the system requires the supervisors' and supervisees' signatures upon completion of supervision visit, and capturing start and end time of the visit, as well as GPS location where the supervision was conducted. The findings further show that the 6 councils that received training from the DUP implementation team documented more planned supervision trips (N=589), compared to those which did not receive training (N=86) from the DUP implementation team. The main challenge noted by this study was untimely supervision visits, delay, or non-delivery of feedback reports to the health facilities from supervisors.