Taking Up the Challenge
President Obama put the rising cost of a college degree in the national spotlight during his State of the Union address January 24. Colleges and universities can take up the president’s challenge to keep tuition costs down by investing in programs, teaching methodologies, services, and support that are proven through a rigorous controlled study to have a positive impact on student outcomes.
Today, decisions about new investments and cuts at universities are often highly political, driven by the interests of various stakeholders, including faculty, alumni, athletics, and administration. Too often, what’s missing is a focus on measurement and accountability, with an eye toward what is most beneficial for students.
Having managed more than 75 controlled studies at universities across the country focused on what interventions do and do not improve graduation rates, I have seen firsthand that this way of thinking about how and where to invest at universities requires a significant paradigm shift.
Perhaps a university has recently launched an online portal for students where they can interact with their professors, look up course materials, and save their work. Or, the school has invested in a rich, week-long orientation program for students, replete with nightly social events, guest speakers, and gift bags for every student. How do we know these efforts are good investments? What are the success metrics for each, and what is the process by which they will be studied, evaluated, and ultimately kept or cut? At most universities, the answer is that data and measurement will not factor into the decision-making process. To improve outcomes for students while simultaneously reducing the cost of earning a degree, this must change.
No doubt it will require focused effort, over many years, to find the right mix of initiatives. But it must begin with a reliable framework for measuring student interventions, so that we can learn what works and what doesn’t.
Take the example of interventions designed to increase graduation rates. Numerous studies are underway to assess the effectiveness of student success programs. Yet only a few of these studies meet evidence standards for effectiveness, such as those defined by the U.S. Department of Education’s What Works Clearinghouse (WWC). Common flaws of these studies include insufficient sample size, a failure to avoid selection bias, a lack of valid or reliable outcome measures, and a failure to assess cost effectiveness. With the right methodology, however, studies will hold up to scrutiny, and the lessons gleaned from them can produce repeatable outcomes.
The sample size issue is more easily overcome at some institutions than others. A small liberal arts college may simply not have enough entering students to conduct a reliable comparison between a control group and an experimental group receiving intervention and support. However, our involvement in more than 75 studies has led us to conclude that careful assessment of the minimum sample size required to yield statistically valid results is required. We must also require evidence from a well-designed study conducted across several universities, or test the intervention with our own student population, before rolling it out to the entire student body—even if that means some students will miss out on the intervention during the testing period.
Measurement with standardized metrics, consistently applied, is the only way to demonstrate replicable outcomes. Equally important, the study must include a cost assessment mechanism to ensure that the intervention is fiscally sustainable and provides the maximum return on the required investment of scarce resources.
Indeed, accountability has never been more important in higher education. That’s why all of us who work in the field, regardless of job title, are increasingly focused on which initiatives demonstrate proven impact and cost effectiveness.
Given our nation’s enormous investment in increasing access to college, and the investment each student makes in starting on the path to a degree, we cannot afford to have students veer off track and fail to reach their goals.
Let’s operate from our strengths as a community, do the necessary research and then act upon it. Let’s collaborate on study design, jointly define metrics, and review our findings together at regular intervals.
This is our moment to begin to build a reliable framework for the measurement of student interventions. Let’s get started now, for the sake of our students and of our institutions.
Kai Drekmeier is co-founder and president of InsideTrack, a provider of student coaching services.