Case Study 3: Austin Peay State University: Degree Compass

Abstract

A case study from Game Changers

Tristan Denley

Students entering higher education face the sometimes daunting task of navigating their way through a degree program. Confronted with a wide array of course options that could satisfy degree requirements, which is the best way to success? In what order should the courses be taken? Course descriptions often give few clues about what the course will entail, containing instead many technical terms that are introduced in the course itself. Advisors are well equipped to provide valuable advice in their own field. But most programs require students to take courses from across the full spectrum of the university, and advisors find themselves challenged to offer useful advice in disciplines far from their own.

All of this assumes that the student has chosen a major that is a good fit. In fact, a sizable proportion of students begin their college career undecided or in a major that they later realize is not what they expected. Complete College America recently reported that students on average take up to 20 percent more courses than are needed for graduation—not because of desire for a diverse curriculum, but because they had to rethink their plans several times. In an environment in which time to degree has considerable implications for a student's likelihood of successfully graduating, a semester of extra coursework plays a crucial factor.

What seemed to be needed was a system that could use the perspective of the past to begin a better-informed conversation between student and advisor. This system would allow advisors and students to make plans for future semesters, equipped with data on courses or even majors in which past students with similar programs, grades, and course histories had found success.

Degree Compass System: How It Works

Inspired by recommendation systems implemented by companies such as Netflix, Amazon, and Pandora, Austin Peay State University (APSU), in Clarksville, Tennessee, developed a course-recommendation system called Degree Compass that successfully pairs current students with the courses that best fit their talents and program of study for upcoming semesters. The model combines hundreds of thousands of past students' grades with each particular student's transcript to make individualized recommendations for current students.

This system, in contrast to systems that recommend movies or books, does not depend on which classes students like more than others. Instead, it uses predictive analytics techniques based on grade and enrollment data to rank courses according to factors that measure how well each course might help the student progress through a chosen program. From the courses that apply directly to the student's program of study, the system selects those courses that fit best with the sequence of courses in the student's degree program and are the most central to the university curriculum as a whole (see Figure 1). That ranking is then overlaid with a model that predicts the courses in which the student is most likely to achieve the best grades. Through this method, the system makes its strongest recommendations for courses that are necessary for a student to graduate, that are core to the university curriculum and the student's major, and in which the student is expected to succeed academically.

Each student's recommended course list is conveniently displayed in a web-based interface on the secure side of the university portal. This interactive interface provides information on each recommended course's curriculum and requirements and what role that course plays in the student's degree program, as well as class availability in upcoming semesters. This same information is also available on PeayMobile, the APSU mobile application (see Figure 2). Faculty advisors can access Degree Compass as a tool for academic advising to supplement the material available to faculty members when they provide advice to their advisees.

Degree Compass also provides a number of enterprise-scale reports that provide strategic information to department chairs and advisors. These reports provide data that enable targeted interventions. For instance, one report allows the institution to enhance its Early Alert System at the outset of the semester by using projected course grades to identify students who would benefit from tutoring support or academic mentoring.

Does It Work?

The main factor in student success and progression lies in the system's ability to place students in courses in which they will be most successful. Faculty and students both welcome the additional information and interact comfortably with the interface.

The grade-prediction model provides an accurate estimate of the final grade a student is likely to receive. When the model's predictions are retrospectively compared with real student grades, we found that 90 percent of the time the model correctly predicted courses in which students would achieve a C or better—on average, it was able to successfully predict grades of C or better to within 0.56 of a letter grade. Moreover, when students' actual grades from their semester courses were compared, grades in courses that were recommended averaged 0.46 of a letter grade better than those in courses the system did not recommend to the student.

Challenges Faced

The main challenge with this system was creating a mathematical model to successfully estimate a student's future grades to an acceptable tolerance, based on the student's transcript and the university's legacy grade data. A secondary challenge was designing a system to sequence courses in a natural order, based on both a given major and the university curriculum as a whole. Once these models were designed and tested, the system then had to be taken to full scale, seamlessly interacting with APSU's course management system.

Course selection is crucial to student success, but so too is the choice of major. The APSU team is currently refining a feature that will allow Degree Compass to suggest majors based on each student's academic record and predicted future grades. We hope that this will be implemented at APSU later this spring.

Can It Work Elsewhere?

As APSU explores replicating Degree Compass at other institutions, the challenges of interfacing with other computer systems and adapting to the curriculum structure of other institutions remain to be fully resolved. Recently, the system played a central role in Tennessee's successful Completion Innovation Challenge application, which received a $1,000,000 award from Complete College America and the Bill & Melinda Gates Foundation to support implementation of Degree Compass at three other campuses in Tennessee—one university and two community colleges. Students, advisors, and administrators at these sister institutions in Tennessee will be able to use the system's features in spring 2012 to create schedules for fall 2012.

One of the major challenges in higher education today is to influence student success, progression, and graduation statistics. If we are to meet President Obama's commitment to having the highest proportion of students graduating from college in the world by 2020, we will need to be able meet this challenge. This system is already making an impact at APSU, and the results from the replications to three other campuses this spring will show how effectively it might be a factor on other campuses. It is our hope that in 2012 we will be able to implement Degree Compass at other universities and community colleges across the nation.

Tristan Denley earned his Ph.D. in Mathematics from Trinity College Cambridge and held positions in Europe and North America before becoming Provost at Austin Peay State University in 2009. His work implements a wide variety of college completion initiatives, spanning pedagogy redesign and the role of predictive analytics and data mining in higher education.

© 2012 Tristan Denley