An Outcomes-Based Program Evaluation of Student U

Abstract

Executive Summary

Program Overview Student U is a small nonprofit organization that offers summer and after-school educational programs to at-risk middle and high school students in Durham, North Carolina. Founded in 2007, the organization has grown rapidly, admitting a new cohort of fifty rising sixth-graders each summer. Student U continues to work with each cohort until high school graduation. The organization is currently piloting a high school program for the oldest cohort of participants, who entered ninth grade in the 2010-2011 school year.

Evaluation Goals and Methods This project provides both a retrospective analysis of past program outcomes and a set of forward-looking recommendations for future evaluation and monitoring activities. It focuses on two separate questions: • Has Student U’s middle school programming achieved intended student outcomes? • What factors identify high school students who are not on track to achieve important program outcomes, such as high school graduation and college enrollment?

Middle School Program Evaluation Data Limitations My analysis uses data on participants’ demographic characteristics and academic achievement levels, measured using North Carolina End of Grade (EOG) test scores. I was able to access longitudinal EOG data for only the first two cohorts of participants. Student records provided by Student U were incomplete, and as a result I was able to obtain EOG data for only 75.5% of participants in these cohorts. Participant Attrition Rates The data show a 17% dropout rate for the first two cohorts of participants. This dropout rate includes all students, not just the 75% of students for whom I was able to access EOG data. Although I find no racial or ethnic disparities between program dropouts and continuing participants, boys were twice as likely to drop out as girls. Interviews with program staff indicate that the students who dropped out likely needed academic support, but were unwilling or unable to commit to Student U’s intensive programming. This high attrition rate may undermine the program’s mission of keeping promising educationally-disadvantaged students engaged with school. Student Achievement To compare participants’ academic outcomes with those of demographically similar non-participating students, I form a non-randomized control group using nearest neighbor matching. A comparison of the treatment and quasi-experimental control group generates no significant effect of program participation on EOG scores. I acknowledge, however, that the small sample size and poor data quality could undermine the validity of my estimates. Moreover, even if the program has no effect on EOG scores, this finding does not indicate that the program fails to achieve any of its goals. Student U also focuses on students’ social development and attitudes toward school, which are not considered in this analysis.

High School Program Benchmarks Policy Review The North Carolina Standard Course of Study defines four high school curricula that students may complete in order to fulfill graduation requirements. Student U participants should complete the college/university prep curriculum in order to be well prepared for postsecondary success. In addition, the University of North Carolina system sets minimum GPA, ACT, and SAT scores for admission to a campus of the university system. The UNC minimum standards, however, are set at extremely low levels. In order to be prepared for college-level academics, students should be encouraged to achieve at levels above the minimal standards. Literature Review My review of academic research summarizes a variety of indicators that predict high school graduation and college enrollment. These factors fall into three general categories: academic performance, educational engagement, and student background characteristics. While SU staff could collect data on many of these indicators, I recommend focusing on those factors most closely tied to academic requirements outlined in the policy review section, and on attitudinal factors that can be measured through Student U’s annual student survey.

Recommendations to Facilitate Future Evaluations Set modest, but sustainable data collection goals. While there are many available indicators of student success, collecting data on all of them will be unsustainable. Student U should establish a few, high-quality measures of student success. These factors include course enrollment, course performance, SAT, and ACT test scores, and surveys measuring student attitudes and confidence about their future educational experiences. Administer consistent surveys and maintain results. Without survey data, SU will be unable to assess its impact on important non-academic goals. Maintaining and analyzing survey responses is a time-consuming task, but Student U’s decision to begin using a standardized, externally generated survey is a promising first step. To facilitate evaluation of individual student performance, SU must maintain survey records in a manner that allows individual student responses to be tracked across years. Monitor attrition rates and explore the causes of student attrition. As outlined in Chapter 2, participant attrition poses a significant challenge for Student U. Depending on the reasons students drop out, this attrition may be a sign that the program is not keeping students engaged with school. Student U should monitor attrition rates, uncover the reasons why students drop out, and take steps to reduce attrition.