By Mikayla Charles, MPH, Program Analyst, Epidemiology, Evaluation, and Metrics, Association of Maternal & Child Health Programs (AMCHP) 

Introduction

Evaluation is the cornerstone for determining a program’s effectiveness, feasibility, and impact from inception to implementation. Evaluations identify successful aspects of a program or intervention while pinpointing areas that require improvement. It is critical to keep program goals and objectives at the forefront of implementing a program and use evaluations to assess the degree to which programs are implemented with fidelity and show evidence that they are working as effectively as intended. The Centers for Disease Control and Prevention (CDC) (2012) defines program evaluation as the systematic collection of information about program activities, characteristics, and outcomes to make judgments about the program, improve program effectiveness, and inform decisions about future program development. Typically, programs are evaluated against performance measures and guidelines tied into a set of evaluation questions to gauge effectiveness. According to the CDC (2012), evaluation questions generally fall into five categories, as shown in Figure 1. Each question is designed to document program progress, demonstrate accountability and effectiveness to funders and policymakers, and identify adaptations needed to ensure a program can sustain positive outcomes. Using the Maternal, Infant, and Early Childhood Home Visiting program as a case study, this article will explore how evaluation drives sustainability in maternal and child health (MCH) initiatives.

Figure 1.

Background

Maternal mortality remains a persistent public health concern. Maternal mortality rates have nearly doubled from 2018 to 2021, escalating from 17.4 to 32.9 deaths per 100,000 live births (The Commonwealth Fund, 2024; Hoyert, 2023). Racial and ethnic disparities amplify this issue: Black women are three times more likely to die from pregnancy-related complications than their white counterparts, and Indigenous Americans are nearly twice as likely to die from these complications (The Commonwealth Fund, 2024; Hoyert, 2023).

Alarmingly, more than 80 percent of pregnancy-related deaths are preventable (CDC, 2023), underscoring the need for MCH programs to lead the way in galvanizing efforts to improve outcomes in this area. MCH programs and initiatives are pivotal in mitigating maternal mortality rates, so it is critical to evaluate these programs and services to ensure they are sustainable and improve health outcomes over time.

In 2023, the Health Resources and Services Administration (HRSA) awarded $90 million for maternal health, including more than $75 million for HRSA’s Maternal and Child Health Bureau (MCHB) programs (HRSA MCHB, 2024a). MCHB hosts various programs to ensure women can access necessary services and care, including the

• Maternal, Infant, and Early Childhood Home Visiting program,
• Title V Maternal and Child Health Block Grant, and
• Women’s Preventive Services Initiative.

In addition, the CDC’s Enhancing Reviews and Surveillance to Eliminate Maternal Mortality program supports maternal mortality review committees in reviewing pregnancy-related deaths to identify prevention opportunities. Many local, statewide, and national organizations focus on MCH by implementing programs and initiatives to combat maternal mortality and address social determinants of maternal health. However, the questions remain:

• Are these programs and initiatives effective?
• Are their outcomes sustainable?

The only way to answer these questions is to focus on sustainability efforts throughout the program evaluation process.

Program Sustainability

Sustainability refers to a program’s long-term effectiveness and impact and is an integral part of the evaluation process. Sustainability pertains not only to the capacity to uphold the program and its outcomes beyond the initial funding phase but also to ensure the program’s ongoing performance and attainment of positive outcomes over time. It is important to integrate performance measures related to sustainability into evaluation plans from the start of a program because achieving sustainability requires careful consideration and adjustment over time and cannot be treated as an afterthought when a program is ending.

Evaluation methods focused on sustainability assess the availability of resources to sustain the program, the support needed to implement the program, and the capacity to continue the work beyond the initial funding period, among other things.

Evaluations should also examine whether the intervention or program has established long-lasting partnerships and collaborations, a sound financial plan, and an organizational structure with the staff capacity to sustain its operations over time (Eval Community, 2024). Evaluations should consider necessary modifications to program or intervention processes, procedures, and implementation plans to align with strategy adjustments. An additional sustainability factor in the evaluation process involves incorporating changes to the program or intervention based on evaluation findings. Evaluations should assess the program’s adaptability to evolve alongside strategy adjustments and its capacity to tackle future challenges.

Case Study: The Maternal, Infant, and Early Childhood Home Visiting program

This case study aims to demonstrate how sustainability is an integral part of the evaluation process.

Overview of the Maternal, Infant, and Early Childhood Home Visiting Program

HRSA’s Maternal, Infant, and Early Childhood Home Visiting program is implemented in partnership with the Administration for Children and Families. The partnership funds states, territories, and tribal entities to develop and implement home-visiting programs to support pregnant people and parents with young children who live in vulnerable communities. Families facing significant risks and barriers to achieving positive and equitable maternal and child health outcomes are urged to participate in the home visiting programs offered by their state. The partnership uses the Department of Health and Human Services Home Visiting Evidence of Effectiveness (HomVEE) review process, which assesses home visiting program models to ensure their effectiveness. Maternal, Infant, and Early Childhood Home Visiting awardees had to select one of the current 24 models meeting HomVEE criteria and tailored to the needs of their community. Therefore, variations in home visiting programs across states and territories resulted. Grounded in research, the Maternal, Infant, and Early Childhood Home Visiting program shows that home visits by a nurse, social worker, early childhood educator, or other trained professional during pregnancy and in the first years of a child’s life improve the lives of children and families (HRSA MCHB, 2024b).

Maternal, Infant, and Early Childhood Home Visiting Program Evaluation

The Maternal, Infant, and Early Childhood Home Visiting conducts annual evaluations at the national, state, and local level for the program. At the state level, each home visiting program undergoes a thorough evaluation to assess the program’s effectiveness. Awardees conduct annual and quarterly reporting on program performance. The Maternal, Infant, and Early Childhood Home Visiting evaluation process uses performance data to monitor progress, pinpoint areas for improvement, and assess whether services have led to tangible improvements for the families served. Evaluation metrics include 19 measures across six benchmark areas:

1. Improved maternal and newborn health
2. Prevention of child injuries, maltreatment, and emergency department visits
3. Improvement in school readiness and achievement
4. Reduction of crime and domestic violence
5. Improvements in family economic self-sufficiency
6. Improvements in the coordination and referrals for other community resources and support

Awardees must demonstrate improvement in at least four of the six benchmark areas to ensure the program is effective for participating families.

Overview of the Sustainability of the Maternal, Infant, and Early Childhood Home Visiting Program

Following a year-long evaluation process of performance measures and reporting, HRSA revised the performance reporting requirements for the Maternal, Infant, and Early Childhood Home Visiting state and territory awardees in 2016. This evaluative process involved soliciting feedback from awardees, federal partners, home-visiting model developers, and other stakeholders. The adaptation aimed to enhance accountability by demonstrating outcomes achieved throughout the program. Additionally, in 2021, HRSA updated the performance measurement system by modifying four of the existing 19 measures and incorporating two optional measures to report on substance use screening and referrals. The performance measures are categorized into two types: (1) performance indicators, which are directly related to the home visiting program alone, and (2) system outcomes, which reflect differences or influences in available system infrastructure at the state or community level (HRSA MCHB, 2024b).

After evaluating each home visiting program at the state or territory level and incorporating input from partners, collaborators, awardees, and other stakeholders, the MIECHV program implemented program adaptations to ensure the effectiveness and feasibility of sustainable program outcomes.

Conclusion

The case study of Maternal, Infant, and Early Childhood Home Visiting programs was chosen to illustrate the importance of sustainability as a fundamental construct of evaluation for several reasons set out below.

• First, the Maternal, Infant, and Early Childhood Home Visiting program represents a widely recognized initiative within MCH programming.
• Second, the Maternal, Infant, and Early Childhood Home Visiting program’s continuity and evolution over time provides context for how to examine the way evaluation processes have contributed to its sustainability. Revised performance reporting requirements in 2016 and updates to the performance measurement system in 2021 demonstrate how the Maternal, Infant, and Early Childhood Home Visiting program evaluation has informed programmatic adjustments to ensure the program continues to be effective and sustainable.
• In addition, the evaluation process requires collaboration and input from awardees, federal partners, model developers, and stakeholders; this program demonstrates the importance of engaging diverse perspectives to sustain MCH programs.

By prioritizing sustainability in evaluation processes, programs can identify areas for improvement, track performance over time, and adapt interventions to meet evolving needs effectively. The sustainability of MCH programs like the Maternal, Infant, and Early Childhood Home Visiting program is vital for addressing maternal mortality and health disparities and fostering long-term health and well-being for mothers, infants, and children across diverse communities.

AMCHP Collaborators

Cheryl L. Clark, DrPH, RHIA, Associate Director, Epidemiology, Evaluation and Metrics, AMCHP, and Meagan Robinson Maynor, DrPH, MPH, Senior Program Manager, Epidemiology, Evaluation and Metrics, AMCHP.

References

CDC. (2012). Introduction to program evaluation for public health programs: A self-study guide. Centers for Disease Control and Prevention. Office of Policy, Performance, and Evaluation. https://www.cdc.gov/evaluation/guide/introduction/index.htm

CDC. (2023). Working together to reduce Black maternal mortality. Centers for Disease Control and Prevention. Office of Health Equity. https://www.cdc.gov/healthequity/features/maternal-mortality/index.html

The Commonwealth Fund. (2024). Policies for reducing maternal morbidity and mortality and enhancing equity in maternal health. A review of the evidence. Commonwealth Fund. https://www.commonwealthfund.org/publications/fund-reports/2021/nov/policies-reducing-maternal-morbidity-mortality-enhancing-equity

Eval Community. (2024). Importance of sustainability as an evaluation standard. https://www.evalcommunity.com/career-center/sustainability/

Hoyert, D. (2023). Maternal mortality rates in the United States, 2021. National Center for Health Statistics. https://www.cdc.gov/nchs/data/hestat/maternal-mortality/2021/maternal-mortality-rates-2021.htm#:~:text=The%20maternal%20mortality%20rate%20for,20.1%20in%202019%20(Table).

HRSA MCHB. (2024)a. Maternal health challenges in the United States. Health Resources and Services Administration. Maternal and Child Health Bureau.  https://mchb.hrsa.gov/programs-impact/focus-areas/maternal-health

HRSA MCHB. (2024)b. Performance indicators system outcomes summary. Health Resources and Services Administration. Maternal and Child Health Bureau.   https://mchb.hrsa.gov/sites/default/files/mchb/programs-impact/performance-indicators-sys-outcomes-summary.pdf