Evaluation Framework

Evaluation Framework for 21st Century Community Learning Centers (21st CCLC) Programs 

 

Introduction

This document has been developed to serve as a basic framework for the evaluation of 21st Century Community Learning Centers (CCLC) programs. This framework was developed by Berkeley Policy Associates (BPA) in collaboration with the U.S. Department of Education, and is based on current standards of practice, evaluation research and the goals of the 21st CCLC program. The Department has contracted with BPA to support its 21st CCLC monitoring efforts, including providing technical assistance to State Educational Agencies (SEAs) for effective evaluations that can be used to support program improvement. This framework provides a basic structure for addressing both the state requirement to conduct a comprehensive statewide evaluation of the programs and activities provided with 21st CCLC funds, and the states’ role in monitoring and supporting evaluation efforts at the local sub-grantee level, as described in the U.S. Department of Education Office of Elementary and Secondary Education’s 21st CCLC Non-Regulatory Guidance:

H-5: State evaluation requirements:

States must conduct a comprehensive evaluation (directly, or through a grant or contract) of the effectiveness of programs and activities provided with 21st CCLC funds. In their applications to the Department, States are required to describe the performance indicators and performance measures they will use to evaluate local programs. State must also monitor the periodic evaluations of local programs and must disseminate the results of these evaluations to the public.

H-6: Evaluation requirements for local grantees:

Each grantee must undergo a periodic evaluation to assess its progress toward achieving its goal of providing high-quality opportunities for academic enrichment. The evaluation must be based on the factors included in the principles of effectiveness. The results of the evaluation must be: (1) used to refine, improve, and strengthen the program and to refine the performance measures; and (2) made available to the public upon request. Local grantees, working with their SEAs, must evaluate the academic progress of children participating in the 21st CCLC program.

As the non-regulatory guidance suggests, at the state level, the focus of the statewide comprehensive evaluation of the 21st CCLC program is on evaluating the effectiveness of programs and activities provided with 21st CCLC funds, and SEAs are also responsible for monitoring local evaluation efforts. At the sub-grantee level, the focus is on assessing progress toward providing high quality services, using evaluation results to support program improvement. At both the SEA and sub-grantee levels, evaluation is to be guided by performance measures, and results are to be made available to the public.

This framework is intended for use by SEAs in support of their 21st CCLC grants. SEA coordinators and evaluators can use this framework to plan or assess the status of their comprehensive state-wide evaluations. SEAs can also use this framework to provide technical assistance and guidance to their sub-grantees in conducting local evaluations. This framework describes five key features of effective program evaluations, and gives examples of how these features are operationalized. It is recommended that 21st CCLC evaluations at both the state and local levels include the following five key features:

  1. Qualified Evaluator
  2. Articulated Program Goals and Measurable Objectives
  3. Design Appropriate for Measuring Program Quality and Effectiveness
  4. Analysis and Reporting
  5. Use of Evaluation Results

Qualified Evaluator

To ensure both the quality and the credibility of the evaluation, it is important that evaluations be conducted by a qualified evaluator, either an individual or team of people with appropriate expertise and experience conducting evaluations of education or afterschool programs. This applies to any evaluation study, whether at the SEA or the sub-grantee level.

  • Qualified evaluators have formal training in research and/or evaluation methods and have previous experience planning and conducting program evaluations.
    • Examples of relevant training include: A Master’s degree or Ph.D. in education or a social science discipline, training in rigorous evaluation design and using relevant qualitative and quantitative methodologies such as conducting interviews and focus groups and/or analyzing survey and administrative datasets.
  • Qualified evaluators have content knowledge of, and experience evaluating or studying, educational programs, school-based programs, and/or specifically after-school programs.
    • Examples of relevant knowledge and experience include: Experience evaluating other 21st CCLC programs or other school or community programs aimed at increasing student academic achievement, experience collecting and analyzing student outcome data (e.g. standardized test scores, grades) and implementation data (e.g., observing classrooms, surveys about program perception, collecting information about program quality).
  • Qualified evaluators are independent of the 21st CCLC program thus avoiding any potential or perceived conflict of interest.

Articulated Program Goals and Measurable Objectives

It is recommended that evaluations explicitly articulate the goals of the program being evaluated and specify how program effectiveness and progress towards program goals are measured.  At the SEA level, program goals align with the overall purposes of the 21st CCLC grant program. At the sub-grantee level, goals and activities are aligned with the state goals but may also reflect local priorities. According to federal statute, the purposes of the 21st CCLC are to:

(1) Provide opportunities for academic enrichment, including providing tutorial services to help students, particularly students who attend low-performing schools, to meet state and local student academic achievement standards in core academic subjects, such as reading and mathematics;

(2) Offer students a broad array of additional services, programs, and activities, such as youth development activities, drug and violence prevention programs, counseling programs, art, music, and recreation programs, technology education programs, and character education programs, that are designed to reinforce and complement the regular academic program of participating students; and

(3) Offer families of students served by community learning centers opportunities for literacy and related educational development.

  • Program goals reflect a “theory of change” or “logic model” which defines the building blocks that are expected to contribute to the long term outcomes. The three broad purposes stated above embody the theory that providing opportunities for academic enrichment, additional youth development and enrichment services, and literacy services to families will result in better academic outcomes for students.
  • While the goals provide the overall theory or logic of the program, measuring success involves identifying measurable indicators for achieving program goals. Effective evaluations explicitly state and incorporate program goals and objectives in all phases of the process including planning, design, and reporting.
  • SEA and sub-grantee evaluations can address the same basic program goals and evaluation questions, or sub-grantees may supplement the state goals with additional goals that are specific to their local needs.
    • Examples of state program goals: Increase students reading skills.
    • Examples of specific sub-grantee goals: improve communication with teachers at host school in order to strengthen linkages between 21st CCLC activities and school day lessons. Increase middle schools students’ academic performance in English language arts.
  • Objectives are specific statements that include measurable indicators for reaching the goals.
    • Example of state goal and  measurable objective:

Goal: To improve student achievement in math.

Objective: To increase the percentage of students participating in 21st CCLC achieving grade level proficiency in math by 10% on the state math assessment

  • Example of sub-grantee goal and measurable objective:

Goal: In a community where violence and behavior are particular challenges, a program goal may be to improve school safety.

Objective: Reduce student disciplinary incidents among students participating in the 21st CCLC program by 15%.

Design Appropriate for Measuring Program Quality and Effectiveness

It is recommended that evaluations use designs that are systematic, well-documented, and measure progress towards achieving program goals and objectives. Designs should be sufficiently rigorous to measure the quality of implementation and to support a reasonable hypothesis that the program is, or is not, contributing to achieving the desired outcomes.

Comprehensive and effective evaluation designs include the following components:

  • Evaluation Questions: Evaluations explicitly articulate the purpose or questions that the evaluation is designed to address.
    • Examples of evaluation questions include: Is the statewide 21st CCLC program reaching the target population? How well are sub-grantee activities aligned with the goals and objectives of the state’s 21st CCLC program?  Is the 21st CCLC program contributing to an increase in reading scores for student participants?
  • Measures: As specified in the non-regulatory guidance, SEAs are required to specify performance indicators and performance measures that are used to evaluate sub-grantee programs. In some cases SEAs may specify a uniform set of performance measures statewide. In other cases, SEAs may want to allow sub-grantees the flexibility to choose between specific performance measurement options, or supplement a core set of statewide measures with additional measures specific to the objectives of their local programs. Comprehensive evaluations include both process and outcome measures.
    • Process measures include measures of implementation fidelity (was the program implemented as intended?), program quality, and program intensity or dosage. Examples of process measures include: program attendance, types of academic or enrichment activities, frequency of these activities, or student/parent/staff satisfaction with the program.
    • Outcome measures are measures of behavior or performance (usually of students) that the program is designed to improve. Examples of outcome measures include: standardized test scores, grades, school attendance records, rates of suspension and other disciplinary actions based on district data.
  • Integrating Process and Outcome Measures: Comprehensive evaluations combine process and outcome measures. Outcome measures identify “what” has been achieved. Process measures supplement outcome measures with information about “how” programs are implemented. Evaluations designed to combine these two types of measures can explore “why” programs may be more successful in some areas than others and what strategies might be effective in addressing program weaknesses. This approach results in an evaluation that is designed to support program improvement.
    • Example of integrating process and outcome measures at the state level: The state evaluator may find that some sub-grantees have shown greater student achievement gains than others. Review of sub-grantees’ Quality Improvement Process reports shows that several sub-grantees with lower student achievement gains have identified the need to increase attendance. Such findings could help the SEA identify a need for TA to sub-grantees on successful strategies for increasing and maintaining high student attendance.
    • Example of integrating process and outcome measures at sub-grantee level: an evaluator may find that reading scores have significantly increased for 21st CCLC participants but math scores have remained stable. Through focus groups, students may reveal that staff members have found ways to make reading groups fun and have created ways to keep students engaged. Such findings could help programs identify successful practices and apply those strategies to math activities, in order to increase student interest and engagement in math. Such information will be uncovered only by asking the right evaluation questions, and linking them to program goals and objectives.
  • Rigorous Design: Using the most rigorous evaluation design that is feasible will provide the best quality evaluation.  Simply reporting achievement on performance measures without some analysis of how the program’s achievements compare to the results that would have been achieved in the absence of the program is not considered to be a rigorous design. Even comparing program outcomes from one year to the next is not considered a rigorous design, if the comparison does not either follow the same group of students over time or control for differences in the characteristics of students from one year to another. The following are examples of different types of rigorous evaluation designs:
    • Experimental (randomized control trial) design: The only way to truly determine causality (if the outcomes achieved are attributable to the program) is through an experimental study using random assignment. In such studies, students or schools would be recruited to (or express interest in) the program and then be randomly assigned to either a program or control group. Experimental designs can be challenging to implement and costly, so they may not be feasible for many grantees.
    • Comparison group designs: quasi-experimental designs compare outcomes between two groups but do not randomly assign individuals to the two groups. Some examples of comparison groups include:

Comparison with district or state averages. This is the simplest type of comparison, and while it does not take into account potential differences between participants and non-participants, it does use district or state averages as a kind of benchmark against which the program can gauge its relative success.

Comparison with a similar group or community. For example, outcomes for adolescents in a Boys and Girls Club in one neighborhood might be compared with outcomes for adolescents in another Boys and Girls Club in a similar neighborhood.

Comparison with matched individuals. For example, comparisons might be made between students involved in a program and students not involved in that program who are matched  to program participants in terms of key variables such as their age, gender, race, grades, receipt of free/reduced lunches, absenteeism, and other  characteristics.

Use of statistical methods to control for measured and unmeasured variables. For example, pre-test and post-test scores for participants can be compared with scores for a comparison group in that school or agency the year before the program opened, controlling for student characteristics.

Regression discontinuity design is the most rigorous quasi-experimental design, but it can be used only under very specific conditions.  If students are admitted to a program based on exceeding a “cutoff” score on a consistent pre-program measure (such as income, test scores, or grade point average), and if an outcome measure is available for both admitted and non-admitted students (those above and below the “cutoff”), this design may be possible.

  • Single group pre-/post-test design: This design is the least rigorous and while it does provide a measure of change for the individual student participants, it cannot be used to infer that the change is due to the program.

[For more information on social science research designs used to evaluate educational programs, see: http://www.socialresearchmethods.net/kb/design.php or  Chapter A6 (p. 201) of The Program Evaluation Standards: A Guide for Evaluators and Evaluation Users, 3rd Edition (2010)]

  • Stakeholder Representation: To produce results useful for program improvement, evaluations collect data from all relevant stakeholders, that is, representatives of all of the key parties who participate in or are directly affected by the program. These include students, teachers, parents, program staff and community partners.
    • Examples of collecting data from key stakeholders: Interviews, focus groups, or surveys of students, teachers, and parents. At the SEA level, an evaluator may interview the state 21st CCLC staff for their input about the program. At the sub-grantee level, an evaluator might interview or survey relevant community partners.
  • Proper Documentation: Evaluations document their designs, methods, sources of data and outcomes. Evaluators should describe the methodology used, data collection strategies and instruments used, analysis plan employed, and any assumptions made. Procedures and methods should be systematic and purposeful.
    • Example: For a sub-grantee evaluation, if 10 students were interviewed at a school about the program, the evaluation should describe how and why those students were selected to be interviewed. Those students should be described (without identifying the individual students); the reader should get a sense of whether those students are representative of other 21st CCLC participants or other students at the school.
  • Data Management: Evaluations use information management and storage procedures that maintain the accuracy of data.
    • Example: Evaluators ensure that data files are backed up; evaluators can have research assistants double enter data for accuracy, all data elements and files are carefully and accurately labeled, all data and artifacts (interviews, documents collected, etc) are securely stored in the evaluator’s office or other safe facilities. Quality control checks are in place to ensure that data are managed and analyzed carefully and accurately. Analysis procedures are documented and accessible to the program or a third party should they be needed for replicating the analysis at a later time.
  • Ethical Standards: Evaluators maintain the confidentiality of participants and use methods and procedures that meet ethical standards.
    • Example: Experienced evaluators are familiar with ethical standards and evaluation participants’ rights in their state and local context.  Students are not individually identified in evaluation reports, and informed consent is obtained if students or parents will be interviewed. For more information, evaluators may visit U.S. Department of Health and Human Services, Office for Human Research Protections at http://www.hhs.gov/ohrp/.

Analysis and Reporting

Data collected is analyzed to answer the evaluation questions, and evaluation reports document both the evaluation methods and results so that findings and conclusions can be clearly articulated and shared with relevant stakeholders.

  • Evaluation reports use data analysis procedures that can statistically determine if an effect is found for program participants.
    • Examples of statistical analyses: regression, analysis of variance, or t-tests accompanied by significant testing to determine whether any differences found are real differences or are due to random error.
  • Evaluation reports include an explanation of how the findings are linked to program goals and evaluation questions.
    • Example of linking findings to state program goals and evaluation: If one program goal is to improve reading skills among student participants, the evaluation report would include a question such as, “Did 21st CCLC program contribute to improvement in reading scores for participants?” The report would then describe how the necessary information was gathered and analyzed. The findings would interpret the analysis to state whether a program effect was indicated.
    • Example of linking findings to sub-grantee goals and evaluation questions: If one of the sub-grantee’s goals is to reduce disciplinary incidents, the evaluation report would include a question such as “How does the number of disciplinary incidents during the current year compare with the previous year?” Then, rather than simply presenting the number of disciplinary incidents, the findings would be presented in terms of whether the goal of reducing disciplinary incidents had been achieved.
  • Evaluation reports describe the characteristics of the sample used to evaluate the program.
    • Examples of descriptions of sample: A statewide evaluation might provide information on how many students, centers, or sub-grantees are in the sample. It might also include information on the demographic characteristics of the students or the size or type of programs (e.g. faith-based organization, school district).
  • Evaluation reports include a description of the data collection methods, including response rates, and sources of information.
    • Example of description of methods: An evaluation that includes a teacher survey would describe the survey instrument, to whom the survey was administered or given, and who completed the survey. It would also provide a response rate (how many surveys were returned and analyzed in comparison to the number of surveys distributed.)
    • Evaluation reports describe any limitations associated with their designs or methods, and their associated limitations in interpreting their findings.
  • Evaluation reports provide recommendations linked to program goals based on findings from the data, including identified strengths and areas for improvement.

Use of Evaluation Results

As mentioned earlier, the non-regulatory guidance requires that sub-grantees use evaluation results to refine, improve, and strengthen their program and to refine the performance measures. Effective use of evaluation results includes:

  • Creating and carrying out an improvement plan based on the findings from the evaluation.
    • Examples of SEA uses of results: identify technical assistance needs of sub-grantees (e.g. strategies for increasing attendance); set academic performance targets for the coming year.
    • Examples of sub-grantee uses of results: identify program needs (e.g., better recruitment of participants); prioritize which academic programs to emphasize to meet academic performance targets in the coming year.
  • Engaging the evaluator in the program improvement process.
    • Example of state level evaluator role in improvement process:  attend management team meetings to consult with the management team on the interpretation and use of evaluation results to identify sub-grantee TA needs and set performance targets for the coming year.
    • Example of sub-grantee level evaluator role in improvement process: facilitate meetings with program staff to engage them in the process of synthesizing evaluation findings and developing action steps.

Illinois State Board of Education. (2021). Framework for 21st Century Community Learning Center Evaluation [PDF]. Retrieved from https://www.isbe.net/Documents/framework_21st_cclc_eval.pdf