Quality Stars NY: Field Test Evaluation Report

Background

QUALITYstarsNY is New York State’s Quality Rating and Improvement System (QRIS). The current QUALITYstarsNY standards were developed to apply to programs and providers under the regulation of one of New York’s public agencies – the State Office of Children and Family Services, the State Department of Education, or the New York City Department of Health and Mental Hygiene.

Like all QRIS, QUALITYstarsNY includes standards for programs and providers based on what is known from research and best practices, procedures for monitoring and accountability, incentives and outreach to encourage participation by programs and providers, ongoing support to help them attain and sustain higher quality of care, and marketing and education with parents to help them use the system in considering care and early education options. QUALITYstarsNY has four categories of standards – Learning Environment; Family Engagement; Qualifications and Experience; and Leadership and Management. Participating programs and providers can be assigned up to 100 points total. The number of points earned will determine a site’s placement in the five-star level system.

Two sets of standards were available for the field test, one for center-based programs and one for family/group family child care homes. During the field test, slight modifications were made to accommodate the inclusion of early childhood programs in public schools. Thirteen (13) locations across New York State were designated as field test communities, reflecting the demographic and geographic diversity of the state.

Overview of Field Test Evaluation

The field test evaluation collected and analyzed data from a variety of sources. In addition to data on the points assigned to the standards checklists submitted by the participating field test programs and providers, the evaluation included:

Observations of classroom and home learning environments, using the Environmental Rating Scales developed by the Frank Porter Graham Center for Child Development at the University of North Carolina (the FCCERS-R for family/group family child care settings; the ITERS-R for center-based care for infants and toddlers, and the ECERS-R for center- and school-based programs for preschool-age children).
Observations of teacher-child interactions related to language and literacy development and social-emotional growth in a sample of preschool classrooms, using subscales from two instruments developed by Dr. Sheila Smith, a member of the evaluation team (the SELA-Supports for Early Literacy Assessment and the SSEGA-Supports for Social Emotional Growth Assessment).
Observations of caregiver-child interactions in a sample of infant-toddler center and family/group family child care home settings, using subscales from an instrument developed by WestEd (the PARS-Program Assessment Rating Scale associated with the Program for Infant/Toddler Caregivers).
Observations of meals and physical activity periods in all settings, using an instrument adapted from one developed by the National Center for Health Promotion and Disease Prevention at the University of North Carolina (the original instrument was the Nutrition and Physical Activity Self-Assessment for Child Care).
Survey questionnaires distributed to attendees at the initial information sessions.
Survey questionnaires submitted by participating programs and providers with their standards checklists.
Survey questionnaires distributed to participating programs and providers attending the post-field test forums held in each field test community.
Notes from structured discussions held at the post-field test forums.
Survey questionnaires mailed to a sample of eligible programs and providers in each field test community.
Telephone interviews with representatives of the lead agencies in the field test communities.

In addition to these instruments, the evaluation team observed field test events such as state and community information sessions; participated in debriefing sessions with observers, quality improvement planning consultants, and lead agency representatives; and scanned many of the documents and materials submitted by participants. The evaluation selected a sample of documents for detailed review and coded information from the quality improvement plans developed by the participating programs and providers and the quality improvement planning consultants.

Key findings from the analysis of data from these sources are highlighted in this summary. Further details on data collection and coding are provided in appendices to the full report.

Key Findings

The full report provides the evidence for the following and other findings from the field test, as well as appendices with detailed information on the evaluation data collection and coding procedures.

Field Test Recruitment, Support, and Participation

A wide diversity of eligible programs and providers in the field test communities applied for the field test, although public schools and family/group family child care home providers were harder to recruit.
A total of 192 eligible programs and providers completed the field test out of the 272 selected for an overall participation rate of 71 percent. Family/group family child care home providers were less likely than centers and public schools to complete the field test (58% compared with 75% and 76% respectively).
No one method of recruitment or support for participation was effective with all programs and providers. Both local and centralized supports were used, as well as group and individualized activities.
Centers and family/group family child care home providers familiar with reporting and documentation requirements were somewhat more likely to complete the field test than those with less experience in those activities.

Measuring Quality with Points Assigned Based on the Standards

Across all participating programs and providers, the average number of points assigned was 45 out of the maximum of 100 possible. Public schools participating in the field test had the highest average number of points (53), in part reflecting the fact that they were assigned 21 points automatically in the Management & Leadership standards category since, because of legal requirements, they were presumed to have met a number of that category’s standards. The average points assigned to centers was 47 and to family/group family child care home providers, 28.
Participants were assigned the fewest points, on average, for the Qualifications & Experience standards category, which accounted for 35 of the maximum 100 points.
Based on the points assigned, on average, almost all public schools and centers were at the 3 or 4 star level, while family/group family child care home provider settings were clustered at the 1 star and the 3 star levels. No field test participants were at the maximum – 5 star – level.
Under scenarios in which the Learning Environment and Family Engagement standards categories were allotted more possible points, and Qualifications & Experience and Management & Leadership fewer possible points, about 20% of the field test participants would have been at higher star levels than under the current point allocation.

Measuring Quality Based on Documentation

The documentation process was considered cumbersome by many participants and in some cases, was believed to be unnecessary or inappropriate. Centers and public schools generally provided documentation in 4 of the 5 areas reviewed for the evaluation: child intake/enrollment, child developmental assessment, curriculum, and family/parent handbook. Few submitted individual staff professional development plans.
Family/group family child care home providers seldom submitted documents related to child developmental assessment or curriculum. Very few submitted budget information.
When points were deducted for inadequate documentation, some participants’ quality (star) level was lowered. Over one-quarter of centers and public schools would have been assigned a lower star level based on adequacy of documentation. Because family/group family child care home providers often did not submit any documentation, and therefore were assigned no points, only 14% of family/group family child care home providers would have been assigned a lower star level based on the documentation review.

Measuring Quality Based on Observations

A total of 327 independent observations of the classroom or home learning environment were made in 188 of the 192 programs and providers completing the field test. These included 52 observations in family/group family child care home settings using the FCCERS, 39 observations in public schools using the ECERS, and 226 observations in centers, 148 using the ECERS and 88 using the ITERS.
Based on these observations of the classroom or home learning environment, participating programs and providers were generally in the adequate to good range on global quality.
Global quality measured by observation was somewhat lower in family/group family child care home settings (averaging 4.5 out of a maximum of 7) than in centers or public schools (averaging 5.0 and 4.8, respectively).
The measure of global quality of the learning environment from the observations was only modestly associated with quality based on the points based on the standards as a whole, with a correlation coefficient of .32 out of a possible 1.0.
Preschool classrooms that were rated high on global quality had areas of weakness in teacher supports for children’s language and literacy development and for their socialemotional growth.
The quality of caregiver-child interactions in center and family/group family child care home settings for infants and toddlers was not associated with the measure of global quality.
Nutrition scores based on observations of the food served in the field test participant sites were not associated with observational measures of global quality or with the points assigned based on the standards. Scores based on observations of opportunities for physical activity also had low correlations with global quality or numbers of points.

Participant Knowledge of Quality and Understanding of the Standards

Almost three-quarters of field test participants learned more about quality as a result of participating in the field test, particularly if they had not reported being involved in prior quality improvement efforts such as accreditation.
While some participants reported having trouble understanding the standards, more were unsure about how to respond to the checklist and what documentation to provide. Standards areas for which one-third to one-half of participants reported difficulty were child observation and assessment, curriculum planning and implementation, family involvement and support, financial accountability and sustainability, and policies and procedures.
Family/group family child care home providers often reported that the standards and documentation requirements for Family Engagement and Management & Leadership did not recognize the nature of their setting.
Participants of all types wanted a clearer link between licensing and regulatory standards and QUALITYstarsNY.
When asked to choose the area for improvement that would help them improve the most, family/group family child care home providers most often selected the purchase of equipment, materials, and supplies for children’s use, while center were evenly divided between similar items and expenditures on training and professional development.
The professional development plans developed by field test participants with the quality improvement planning consultants focused on improvements in curriculum, adult-child interactions, and knowledge and skills in teaching and learning, all areas that would directly address the Learning Environment standards.

Climate in the Field

Many programs and providers in the field test communities were unaware of QUALITYstarsNY. Even those who attended information sessions reported being largely unfamiliar with the standards and process.
About half of field test participants would definitely participate in state-wide implementation of QUALITYstarsNY. Many programs and providers, regardless of their involvement in the field test, believed that at least half of those eligible would participate once QUALITYstarsNY becomes fully implemented.
Gaining information about quality and having access to supports for quality improvement were major benefits expected for participants in state-wide implementation of QUALITYstarsNY.
The amount of time and effort required, particularly in providing documentation, were concerns for many participants and others in the field test communities.
The majority of participating programs and providers of all types believed that statewide implementation of QUALITYstarsNY was very important. Field test participants and lead agencies believe that the value of QUALITYstarsNY in improving children’s and families’ access to high quality early care and education is a message appropriate for all stakeholders.

Recommendations for State-wide Implementation from Participants

Field test participants and the lead agencies had a number of specific recommendations related to state-wide implementation of QUALITYstarsNY. These included:
o Clearer information about the process and requirements for participationo More explicit rationale for the standards and documentation requirements as they apply in different settings (centers, public schools, and family/group family child care homes)o Alignment of the standards with other standards and reporting requirementso Alternative ways to demonstrate compliance with the standards and/or to provide documentationo More guidance and feedback on adequate or acceptable documentationo Assistance in understanding and preparing for observations and for the quality improvement planning consultations
Resources for quality improvement were the incentive most often mentioned for encouraging and rewarding participation in QUALITYstarsNY. Most field test programs and providers were not able to draw on their own funds to support professional development.
Local communities often lack the capacity to provide more intensive, ongoing, and individualized support for quality improvement. Resources for obtaining appropriate materials and equipment are also scarce.

Implications for State-wide Planning and Implementation

A number of implications for planning state-wide implementation of QUALITYstarsNY are provided throughout the report. This summary draws together and highlights the major implications.

Standards – Content

The QUALITYstarsNY standards categories in the areas of learning environment and family engagement are better understood and accepted by field test participants. It will be important to make sure that the rationale for all the standards is clearly communicated.

Standards – Applicability in All Settings

Family/group family child care home providers and public schools were less comfortable with how the standards and in particular the terminology and documentation requirements applied to them. Consideration might be given to alternative language and recognition of alternative ways of demonstrating quality.

Standards – Coverage of Key Dimensions of Quality

Key dimensions of quality known to be closely tied to children’s development and learning do not appear to be fully captured in the current standards and measurement tools. These dimensions include the quality of adult/child interactions and relationships, use of intentional strategies to promote language and early literacy development, and provision of good nutrition and opportunities for physical activity.
Consideration of additional standards needs to ensure that they are well understood and accepted in the field and that they can be reliably and accurately measured.

Standards – Comparability with Other Performance/Reporting Requirements

Development of the standards drew upon licensing requirements, national program standards, and accreditation processes. Finding ways to align reporting and documentation requirements would avoid duplicated effort and encourage participation.

Documentation Requirements

The number of individual documents requested and the time and resources involved in submitting documentation were barriers to full participation, particularly by programs and providers not familiar with similar requirements. Identifying a core set of documents for submission, allowing for on-site view of some documents, and accepting other ways of demonstrating that the standards are met could be ways to address this barrier.
Some programs and providers did not understand what documentation would be appropriate; others submitted documentation that appeared inadequate. Providing a set of rubrics or essential elements for core documents would help prepare participants for success in meeting the standards.
Providing information necessary for the Qualifications and Experience standards was onerous for many participants. Having a state early childhood education workforce registry would alleviate this paperwork burden as well as resolve concerns about providing personal information.

Observational Tools

More education and training on any observation tools used in QUALITYstarsNY – whether the ERS or others – will be needed to build confidence in their use, help programs and providers prepare for their administration, and develop quality improvement strategies that address the dimensions of quality covered.
While the process of conducting and providing feedback on the ERS observations will not be constrained by evaluation requirements, unions and other groups may have concerns about how the results will be shared and used by supervisors, program administrators, or licensing agencies.
As noted above, there are aspects of quality that are not fully addressed by the ERS instruments, or these instruments may not help identify specific areas for improvement. If additional observations on these other aspects of quality are included in QUALITYstarsNY, consideration will need to be given to how to do so efficiently. For example, should additional observations be made after programs and providers reach a basic level of quality, or are they so critical to improving children’s learning experiences that they should be incorporated earlier in the process?

Submission Procedures

Many programs and providers used on-line tools to apply for the field test and many recommended this option for submitting the standards checklist and documentation. Continuing to build capacity to do this will be an important infrastructure development.
The desire by many participants to have an iterative submission process with feedback would require greater effort in tracking and reviewing submissions. Consideration might be given to having preliminary reviews at the local or regional level before formal submission to QUALITYstarsNY.

Point Assignment

As noted above, many documents that were submitted during the field test appeared inadequate to meet the intentions of the standards. In addition to providing participants with a list of essential elements or a set of rubrics for core documents, a process for reviewing and assessing submitted documents will be needed.
There was interest in being able to achieve partial points if some but not all components of a specific standard were met. A related concern was that, for some standards, the points were contingent on previous documents in ways that participants did not feel allowed them to get credit for what they did do. (An example is that no points could be given for submitting a parent handbook if other parent communication activities were not also documented.) This aspect of the current standards was confusing and might be reconsidered.
Participants who did not currently serve infants, children with special needs, or children whose home language is not English believed that not being assigned points for meeting the needs of these children was unfair. Some indicated that they had the capacity and willingness to serve these children once enrolled, but there was no way to demonstrate this in the standards checklist.

Weighting of Categories

The current weighting of the standards categories raised considerable concern during the field test. Many participants of all types wanted greater weight given to the learning environment and family engagement categories.
Using alternative weighting that gave greater importance to these two categories allowed family/group family child care home providers in particular to demonstrate higher quality. This raises the question of whether different weighting might be appropriate for different settings.

Recruitment of Centers, Public Schools, and Family/Group Family Child Care Home Providers

As noted earlier, more education on the importance of the standards and their relevance to different programs and providers will be important in marketing QUALITYstarsNY. In addition, requirements, supports, and benefits can now be more clearly described to potential participants.
The special concerns of family/group family child care home providers will need to be addressed. For example, these providers may especially need reassurance that participation will result in greater benefit than harm, given the general pattern of these settings receiving lower ratings than centers and public schools.
Multiple outreach strategies with follow-up will probably be needed in the initial stages of state-wide implementation. Working through local or regional organizations would make use of their existing relationships with programs and providers. Strategies to build new relationships, for example with public schools, may also be needed.
Funding for improvements, especially the purchase of materials and equipment, and opportunities for professional development would be attractive incentives.

Support for Participation and Quality Improvement

Training on the standards and on the observation instruments, as mentioned earlier, will be helpful in recruiting programs and providers. Such training should also increase the submission of complete and appropriate materials.
Some programs and providers that wish to participate in QUALITYstarsNY will need little assistance in compiling and submitting the necessary materials. However, others would benefit from some support. Offering a menu of supports – through group sessions, individualized consultation, webinars, peer groups – would be most likely to meet the range of needs and interests.
As noted earlier, having an iterative process for submission, review, and feedback is another way to support programs and providers entering QUALITYstarsNY.
Individualized on-site consultation was generally greatly valued by the field test participants. Ways to improve the benefits from this consultation could include developing quality improvement plans with specific steps over a period of time toward long-term goals, making more information on the results of the rating process to the programs and providers in advance of the consultation, providing follow-up contacts by consultants to help support and fine-tune implementation of improvement plans, and building relationships among participants in the same community to provide peer support and mentoring.

Marketing to Other Stakeholders

Messages to all stakeholder groups should highlight the demonstrated link between experience in high quality early care and education settings and children’s development and learning.

Executive Summary

Download PDF