Scenario

With more than 100 students in his intro to psychology course, Dr. Delgado is unable to provide the level of individual attention for each student that he would prefer. Inevitably, some highly capable students drop out or do poorly, in part, he believes, because they simply need extra guidance. Delgado signs up for a pilot of a learning analytics (LA) program at his institution, and he describes to students how the system works to try to preempt concerns they might have about it. Under this program, a computer application keeps track of students’ digital activities—how often they log in to the LMS, which online resources they download, the number of posts they make on blogs and course discussion boards—and compares these indicators of activity with other factors, including class attendance, grades on assignments and tests, and peer evaluations, which have long been a part of Delgado’s courses.

The course progresses, and the amount of data collected balloons. The LA system runs weekly analyses, looking for associations between the variables and outcomes. When it spots students whose activity suggests a low likelihood of success, it automatically notifies them and suggests steps they could take to increase their odds of success in the course. Some students disregard the notices, but others heed the advice and increase their engagement. Delgado also receives notifications, and he personally contacts several students identified as at-risk. These students appreciate the extra attention and make changes to their level of participation and activity in the course. For students doing well, the LA reports provide needed reassurance that they are likely to succeed in the course. For many students, this course is one of their first experiences in college, and the LA system serves as an objective gauge of how they are doing.

Midway through the term, Delgado sees in the LA reports that during a two-week unit on psychological disorders, measures of student activity drop considerably for most students. Quiz grades don’t change much, but the level of engagement between students and with the readings is off sharply. In talking with other faculty and his TAs about the change, Delgado speculates that many students are reluctant to talk openly about experiences they have had with mental illness, particularly among family members. He devises a set of exercises in which students can anonymously participate in such online discussions, and participation spikes.

1. What is it?

Broadly speaking, analytics programs evaluate large data sets to provide decision makers with information that can help determine the best course of action for an organization. Learning analytics (LA) takes this same approach but with the specific goal of improving learning outcomes, which could be measured by grades, retention, or completion. LA collects and analyzes the “digital breadcrumbs” that students leave as they interact with various computer systems to look for correlations between those activities and learning outcomes. LA software compares a student’s activity with others in the class, with students who previously took the course, and/or against other rubrics to create a model for how each student is likely to fare. In this way, LA capitalizes on the vast quantities of data that most colleges and universities collect to find patterns that can be used to improve learning.

The most common use of learning analytics is to identify students who appear less likely to succeed academically and to enable—or even initiate—targeted interventions to help them achieve better outcomes. At the same time, some faculty employ LA tools to identify specific units of study or assignments in a course that cause students difficulty generally. Instructors can then make curricular changes or modify learning activities to improve learning on the part of all students.

2. How does it work?

Much of the data on which LA applications depend comes from the learning management system (LMS), including log-in information, rates of participation in specific activities, time students spend interacting with online resources or others in the class, and, in some cases, grades. Applications that perform data collection and analysis are frequently either built into or added onto the LMS from which they draw primary data. LMS vendors increasingly offer analytics tools tied to their software, while other LA applications are built by colleges or universities or by third parties to work with the LMS.

LA applications gather data, analyze that data, generate reports, and enable interventions. In most cases, this happens without an opt-in by students. The type of data gathered varies by institution and by application, but in general it includes information about the frequency with which students access online materials or the results of assessments from student exercises and activities conducted online. The types of analyses performed vary, but one approach involves the evaluation of historical student data to create predictive models of successful and at-risk students. Reports can take various forms, but most feature data visualizations designed to facilitate quick understanding of which students are likely to succeed. LA applications can provide this information to students, faculty, or both. Some systems proactively notify users; other systems require users to take some action to access the reports. System-generated interventions can range from a simple alert about a student’s likelihood of success to requiring at-risk students to take specific actions to address concerns.

3. Who’s doing it?

Some for-profit institutions have been among the early adopters of learning analytics, with complex systems put in place early at institutions like the University of Phoenix, Capella University, and the American Public University System (APUS). The analytics software at APUS, for example, pulls data daily from an extensive data warehouse, compares it using statistical measures and semantic analysis engines, and presents results in a visual format. Among other routine analyses, the entire student body is ranked each week according to probable success with coursework. Interventions are then executed for those at greatest risk, and aggregate data are evaluated to see whether changes in the curriculum are necessary.

Traditional higher education institutions like Rio Salado College in Arizona have also begun to implement LA and predictive modeling. Because the college found that early grades in coursework were not a reliable factor in predicting student success, its LA software focuses on information such as log-in frequency, the pace of a student’s work in the first eight days of class, and student involvement in discussion forums. In other LA systems, grades are part of the analysis: At Louisiana State University, LA tools factor in grade data; where grades are high but a student’s participation is low, the software ignores the concern, sending out a caution only when both are low.

4. Why is it significant?

Learning analytics tools can track far more data than an instructor can alone. For high-enrollment classes in particular, LA provides a support structure that monitors student progress and offers timely guidance to students whose academic underperformance might otherwise go quietly unnoticed. At their best, LA applications can identify factors that are unexpectedly associated with student learning and course completion. This information can be valuable both to students, in how they approach learning, and to faculty, in how they structure curricula. With the help of analytics tools, students and instructors can better understand the learning process and take action to improve course outcomes.

5. What are the downsides?

Any application that accesses and analyzes student data must comply with privacy regulations, which adds cost and complexity. Moreover, the behind-the-scenes monitoring of so many aspects of students’ daily activities raises the specter of a “digital Big Brother.” For their part, some instructors bristle at the notion of a computer system that catalogues and evaluates all online activity of the students in their courses and, in doing so, appears to encroach upon faculty authority and autonomy. LA works best in courses that feature regular grades, extensive activities, or other points of evaluation, and faculty who teach “midterm-and-a-final-exam” courses might feel some pressure to redesign those courses. Students and instructors alike might feel as though LA takes assessment out of the realm of human judgment and reduces it to numbers and statistics. Finally, data for LA applications can come from divergent sources and often must be converted to consistent formats and structures to allow proper analysis.

6. Where is it going?

Graphical presentations of analytic data are already moving toward more compelling, dynamically generated views available in real time to faculty, advisors, and students. As new types of data are added—from clickers, social media sites, or even cloud-based applications used in coursework—the systems will become more sophisticated and more robust in their ability to model and predict student success. Data gathering across such an environment allows a student’s probability of success to be measured against a broader base, resulting in more reliable assessments of the interplay of factors that are associated with different learning outcomes and in more effective strategies for helping at-risk students get back on track to succeed.

7. What are the implications for teaching and learning?

The goal of first-generation learning analytics is intervention for at-risk students, in time to help them succeed. Often students who are having difficulties are not certain what action they should take or how they should approach their instructors. A simple heads-up from these systems can provide a starting point for helpful conversations with teachers and advisors. Beyond this focus on individual students, LA can provide data to instructors, who can gauge whether a class understands the concepts in the coursework and make changes in real time to remove roadblocks to learning. In the area of institutional accountability, LA tools can leverage information that is imbedded in student records to show that student monitoring is done on a routine basis and that the institution has consistently taken appropriate action to support student efforts.

December 2011

© 2011 EDUCAUSE
This work is licensed under a Creative Commons
Attribution-NonCommercial-NoDerivs 3.0 License.
http://creativecommons.org/licenses/by-nc-nd/3.0/

educause.edu/eli

EDUCAUSE 7 Things You Should Know About…™

EDUCAUSE is a nonprofit membership association created to support those who lead, manage, and use information technology to benefit higher education. A comprehensive range of resources and activities are available to all EDUCAUSE members. For more information about EDUCAUSE, including membership, please contact us at info@educause.edu or visit educause.edu.