State Indicators for Early Childhood: Short Take No. 7

Learn about our Project Thrive project.

Virtually all State Early Childhood Comprehensive System Initiatives have adopted or identified indicators for monitoring program performance and child outcomes related to early childhood systems. These are primarily based on nationally recommended indicators or on state initiatives. However, although a functional set of indicators is needed to monitor progress of ECCS initiatives across the states, there is no one overarching set of indicators consistently being used. The challenge for states’ ECCS leadership is to select an indicator set that is both comprehensive enough to monitor system developments and specific and limited enough to be useful and manageable. This Short Take reviews the characteristics of good indicators and proposes 36 indicators, based on a review of the literature, an analysis of key national indicator sets, and a comparative review of indicators set out in State ECCS reports and plans.

Background

In recent years, states and communities across the country have adopted results-based accountability (RBA) approaches that use indicators of child and family well-being, including those focused on early childhood health and development. Federal agencies have defined program performance measures, and federal interagency efforts have created indicator sets. As defined by Friedman and others, a full RBA process has multiple, cyclical steps to:

1. convene stakeholders;
2. select desired results and indicators;
3. gather baseline and trend data;
4. investigate the story behind the baseline and trends;
5. learn about effective intervention strategies (“what works” to do better);
6. recommend and implement a change strategy based on knowledge and consensus; and
7. repeat the cycle.

Some communities, states, and federal agencies are actively engaged in the work of monitoring the well-being of children and families using the full RBA process.

For others, indicators and performance measures are used to underscore areas of need or success. As described by Lisbeth Schorr:

The use of outcome indicators that reflect common sense and common understanding (indicators that show, for example, that the rates of low-weight births are being reduced, or that more students are demonstrating age-appropriate mastery of school skills) helps to focus attention on agency mission rather than rules, and permits the necessary flexibility and autonomy at the front-end… The focus is on results and performance, not just on the number of clients served or encounters. The question asked of agencies and service providers, shifts from “Did you do what they told you to do?’ to “Did it work? What difference did it make in outcomes for children?

For RBA efforts, a desired result must be determined. Here, a “result” is a “bottom line” condition of well-being for children, families, or communities. For State ECCS initiatives, the desired results that all children reach school healthy and ready to succeed, with age appropriate social-emotional, cognitive, and language development. From a policy perspective, these are reflected in the core ECCS goals and five core components: (1) access to health care and medical home; (2) social-emotional development and mental health; (3) early care and education; (4) parenting education; and (5) family support.

In RBA terms, an “indicator” (sometimes called a benchmark) is a measure (supported by reliable and routinely available data), that helps to quantify the achievement of the desired result. While these measures don’t replace evaluations of what works, monitoring with indicators helps to answer two important questions: (1) “How would we know if we achieved the desired result?” and (2) “Are we making progress, moving in the right direction toward the desired result?” For ECCS Initiatives, with the overall desired result being school readiness, a good set of indicators would help to answer these questions: What are the measures of school readiness that would indicate success? What are the indications that a community or state is engaged in actions that increase children’s school readiness?

An indicator can measure risk, process, or outcomes. The most important indicators measure the outcomes of a population as a means of tracking progress toward desired results. Some are measures of policy implementation, with others focused at the individual level. A particular indicator may measure risk in some situations and at other times be an outcome indicator. The low birthweight rate, for example, would be an outcome when considering prenatal programs and a measure of underlying risk in terms of outcomes for children enrolled in early childhood programs. Fourth grade reading scores would be an outcome measure for early childhood programs, and might be an indicator of risk if the focus is having all children graduate from high school.

Project THRIVE has worked in partnership with states to define appropriate indicators for ECCS initiatives. We have adopted the three Friedman criteria used to identify an effective indicator.

These are:

• Communication Power – Does the indicator communicate to a broad range of audiences? Is it easily understood by the lay public, policy makers, and media? Does it measure or reflect something that is widely understood as a problem or remedy?
• Proxy Power – Does the indicator say something of central importance about the result? Does the indicator also reflect associated factors and risks? For example, infant mortality is an indicator considered to be a reflection of maternal and infant health, of premature or low-birthweight birth, and of access to a healthy environment and high quality health services.
• Data Power – Are data routinely available on a timely basis? Are these data reliable and standardized? Do we have both a valid numerator and denominator for calculating rates? If not, is the need for this indicator feasible and important enough to be put on the agenda for data development?

Understanding and Mining Existing Data Sets

Project THRIVE conducted an assessment of early childhood system indicators based on a review on the State ECCS reports and plans from 2006 and 2007. Then, using an iterative process, we created a comprehensive matrix of indicators, including relevant measures from several key national indicator sets and State ECCS efforts. The indicator sets reviewed included: the Title V national performance measures, measures proposed by the Institute of Medicine, and the School Readiness Indicators datasets highlighted below. Additional early childhood policy and program measures from the NCCP “Improving the Odds” project also were added to this matrix. This yielded a list of more than 200 indicators. We considered each indicator in light of data power, communication power, and proxy power, as well as relevance to the five core components of ECCS.

Federal Maternal and Child Health Measures

Maternal and child health programs have a long history of experience with monitoring the health and well-being of children and families, stretching from the program’s origins in 1912 to today. In response to the mandates of the 1993 Federal Government Performance and Results Act (GPRA), the Maternal and Child Health Bureau (MCHB) and the states developed a set of nationally mandated Title V performance measures, which continues to evolve. This federal structure creates an accountability and performance monitoring framework for State Title V Maternal and Child Health Programs. Each state is required to report on a core national set of measures, which includes three types of measures, including: (a) 18 performance measures; (b) six outcome measures, and nine health systems capacity measures. With the exception of indicators related to infant mortality, none is specific to Birth to Five, or responsive to the ECCS vision and system goals.

Indicators from the Institute of Medicine/National Academy of Sciences

Through several publications, the Institute of Medicine/National Academy of Sciences Board on Children and Families has reported on indicators and performance monitoring. One important study, Children’s Health, The Nation’s Wealth: Assessing and Improving Child Health, reviewed the data and methods used to monitor children’s health and wellbeing, using a broad definition of health. This collection of more than 100 indicators “highlights the types of measures used in national initiatives that present indicators on various aspects of children’s health and may provide a useful reference for states and localities interested in developing state or local indicators.” While most of these focus on the whole of the childhood years up to the age of 18, many can be adapted for use relative to the birth-to-five age group. The measures in these reports are based on both primary and secondary data collected from state and federal agencies, thus data are routinely and reliably available for many of the indicators. A smaller number of indicators are proposed for which data are not routinely available at the national, state, or community level.

School Readiness Indicators Project Measures

The School Readiness Indicators (SRI) project aimed to develop child well-being and development indicators to improve school readiness and ensure early school success using an ecological model starting with the child and family, and moving to the school and the community: This project brought together senior policy and data staff from multiple public agencies within 17 participating states, as well as national experts. Together, they selected a common set of indicators linked to state priorities and policy options. A multi-dimensional concept of school readiness with strong health-status and health-systems components, this set of 23 core indicators uniquely focuses on children from birth through age eight. This set of indicators aims to measure school readiness in children, families, communities, health services, early care and education, and schools. The report also includes “emerging indicators” which are deemed to be critically important but currently are difficult to measure and track at the state level.

State ECCS Indicators

Under ECCS grant guidance, states are charged with building partnerships with other stakeholders and developing statewide comprehensive plans including indicators. States are at various stages in the process of selecting and using indicators, with some having adopted an RBA framework in which indicators are being used to guide government decisions at the state and local level, and others are in the early stages of selecting indicators. Notably, some states are using indicators for which data would not be available in all states (such as, school readiness in all domains based on kindergarten assessment, or child care programs with mental health consultation available).

Figure 1 lists Project THRIVE’s recommended indicators, according to over-arching categories and the ECCS five core components. These are illustrated in Figure 2.

Developing ECCS Indicators

Based on the Project THRIVE analysis, a short list of 36 indicators was identified and is shown in Figure 1 and illustrated in Figure 2. This set of proposed indicators includes outcome, process/program/policy, and population risk measures. Where data are available, states, and even city/county areas, could monitor some main risk factors that affect need for early childhood interventions, assess performance of key programs and policies that affect school readiness, and measure key outcomes. This list has been discussed by State ECCS leaders through conferences, webinars, and an online discussion forum.

Box 1 highlights some promising state-level ECCS indicators not found in national sets. The boxes also include examples of state strategies for implementing the use of indicators, suggesting the variety of ways of approaching the challenge. It should be noted that one domain where the need for indicator development is particularly great is around family support.

Conclusions and Recommendations

Based on our review and analysis of these indicator sets, we made the following observations:

• At this point, there is no single, commonly agreed upon set of indicators to measure and track early childhood systems and school readiness. This should be a priority for ECCS, with data development efforts to support use of key indicators for which data are not now available in all states.
• Only a small number of states are conducting school entry assessments that would permit population-wide measures of “school readiness” across domains of development. At the same time, this is a measure with potential validity (based on school entry assessments), the clearest “outcome” measure of programs and policies designed to improve school readiness, and important to support new policies and continued investment. This should be a high priority for data development in all states, and federal funds should be allocated to support such efforts.
• Indicators relating to medical homes are included in some state lists but primarily as an indicator for which data development is needed. Functional definitions and routine, reliable data sources were not found in any state.
• Indicators relating to mental health and social-emotional development are sparse. States have proposed several, largely related to access to or utilization of mental health services and/or early childhood mental health consultation. A few states are taking early steps to monitor screening for maternal depression or social-emotional delays in young children. An opportunity exists to build on national survey data measuring preschool expulsion.
• Similarly, indicators related to parenting education and family support are lacking. For example, regular reading to young children is an important measure with predictive value. Unfortunately, the state-by-state data for this indicator is available only every few years in the National Survey of Early Childhood Health. Beyond that, additional work is needed to develop a broader consensus about important programs and indicators in this area.
• There is only one state (Vermont) that we are aware of that uses an indicator that measures the documented synergistic effect of a combination of risk factors. The NCCP Improving the Odds project (www.nccp.org/projects/improvingtheodds.html) has collected data on the percentage of children with multiple risks by state. This measure includes having parents that: (1) are single-heads of household, (2) live in poverty, (3) do not speak English well, (4) have less than a high school education, and/or (5) have no paid employment (many states include several of these as individual indicators).
• In choosing and formulating their indicators, some states appear to be struggling with understanding the definition of a good early childhood health and development indicator. Some have confused goals and objectives with indicators. More peer-to-peer learning and technical assistance could help remedy this problem.
• Only a few states have actually moved from identifying indicators to use of this information in a results-based accountability process. Examples in states such as Michigan and Vermont provide insight into ways to use indicators to advance ECCS work at the state and local levels.

Recommendations

• Given the focus on early childhood development and school readiness in every state, there is an immediate need to develop consensus and data to support one to three key outcome measures that could be used across states and across federal agencies. We have proposed three, that ideally would also linked in federal performance measures across systems:
  — fourth grade reading achievement,
  — readiness in all five domains,
  — undetected development delays or chronic health problems at school (kindergarten) entry.
• Each State ECCS initiative should review its own indicators and consider ways to enhance data collection and their ability to work with data, and move toward greater use of RBA.
• The identification of a common set of performance measures across federal agencies involved with early childhood, starting with the three above, would help the states move forward. This work might build on the groundwork laid by the Federal Interagency Forum on Child and Family Statistics. They should address differences in risk, address the five ECCS domains, reflect the outcomes of cross system work, and take account of our knowledge of effective policy and practice.
• There is a need for federal incentives, including legislation and fiscal incentives, to support an interagency data development approach, linked with a core set of indicators (perhaps 8-10).