Chapter 4: From Metrics to Analytics, Reporting to Action: Analytics’ Role in Changing the Learning Environment

Abstract

A chapter from Game Changers

Linda Baer and John Campbell

This chapter is licensed under a Creative Commons Attribution 3.0 Unported License

What will the game-changing tool kit look like for next-generation learning? How can institutions prepare to meet the increasing demands? Institutions will be required to transition from metrics to analytics and from reporting to actionable interventions. In this next generation of the learning environment, analytics will play a role in higher education. But leading the institution from metrics to analytics and reporting to action will require a significant institutional shift.

Setting the Context

A renewed sense of urgency for improving higher education's accountability, transparency, and performance is in place—the result of a perfect storm of state budget challenges, the ongoing transition from a manufacturing to a knowledge economy, and the inability of the value of higher education to be appropriately articulated. Students, parents, accreditation agencies, and other external constituencies are demanding more from higher education, searching for an overall return on this investment from the student, state, and federal perspective. Issues requiring attention include increasing degree completion and decreasing the achievement gap, as well as changing the focus from access to success and from seat time to competencies. As with all aspects of learning, these challenges cannot be met with simple changes. Institutions must strive to develop analytics or "actionabl e intelligence" in all institutional areas—particularly in learning.

Higher education has access to more data than ever before. Technological tools and resources are strengthening the institutional capacity to access data to improve decision making. Smarter tools that are leading to adaptive learning and personalized opportunities will soon be a reality. In fact, analytics on institutional data will prove key to transforming student retention, graduation, and success.

Performance metrics on student learning, progression, and completion are becoming more prevalent across the country, driven by efforts from the White House, statehouses, accrediting agencies, and local communities.

• The American Graduation Initiative, proposed by the Obama administration (but not passed), called for states and colleges to "establish quantifiable targets for improving graduation rates" in order to access available federal funds.¹
• The Obama administration seeks to increase the number of college graduates by 5 million by 2020. The administration believes this is necessary to rebuild the capacity and competitiveness of America's workforce.²
• Twenty-nine states have joined the Complete College America Alliance of States to develop specific plans to improve college completion rates.³
• The National Governors Association targeted the Complete to Compete initiative, which focuses on increasing the number of students in the United States who complete college degrees and certificates.

According to Complete College America, nearly one in two students pursuing a bachelor's degree will not obtain that credential within six years, and fewer than one in three will complete a two-year college degree in three years. Sixty-two percent of jobs will require college education by 2018, and more than half of those will require at least a bachelor's degree.⁴

In Education Pays, Baum and Payea describe the value of a college education. Higher education continues to help people attain success both socially and economically. In addition, college graduates experience a host of other benefits from a college education, such as increased earnings, increased voting behavior, lower rates of incarceration, and higher rates of good health and charitable activity.⁵

A college degree nearly doubles annual earnings. The report entitled The Big Payoff ⁶ reveals that over the course of an adult's working life, high school graduates can expect, on average, to earn $1.2 million; those with a bachelor's degree, $2.1 million; and people with a master's degree, $2.5 million. Persons with doctoral degrees earn an average of $3.4 million during their working life, while those with professional degrees earn the most, at $4.4 million. This additional income will fuel the national economy and raise the standard of living.⁷

Yet, results from a recent report by Complete College America entitled Time Is the Enemy ⁸ indicate that

• 75 percent of today's students are juggling family, jobs, and college while commuting;
• part-time students rarely graduate;
• poor students and students of color struggle the most to graduate;
• students are taking too many credits and too much time to complete; and
• remediation is broken, producing few students who ultimately graduate.

With the national, state, and local calls for more accountability comes the need for institutions to develop more data capacity and to optimize student retention and completion. According to Bailey et al. in Unleashing the Potential of Technology in Education, "We are at the dawn of an era in which educators have the potential to harness technology to produce a step change in student achievement. Although visionaries have been promising for years that technology would transform primary and secondary education—and despite the billions of dollars spent on networking schools and equipping them with computers and other devices—the actual impact on student outcomes to date has been disappointing. Yet when technology is strategically introduced into every step of the educational value chain, it does, in fact, have the potential to enhance every aspect of instruction and learning." In order to dramatically improve student outcomes, technology must be fully aligned with educational objectives, standards, curricula, assessments, interventions, and professional development.⁹

Emergence of Analytics: An Evolution of Enterprise and Instructional Systems

The interest among higher education institutions in analytics has grown since early projects impacting student success were highlighted by Campbell, DuBlois, and Oblinger. In their 2007 article "Academic Analytics," the authors cite that institutions' response to internal and external pressures for accountability in higher education, especially in the areas of improved learning outcomes and student success, will require IT leaders to step up and become critical partners with academics and student affairs. They argue that IT can help answer this call for accountability through academic analytics, which was emerging as a critical component of the next-generation learning environment.¹⁰

As the interest in academic analytics in higher education has grown, so have the escalating accountability demands that are driving performance measurement and improvement in interventions. Improving performance will require coordinated measurement, intervention, and action across the entire education/workforce spectrum—from "cradle to career."¹¹

There is a wide continuum of activities within the ecosystem of analytics. As Phil Long and George Siemens relate: "Analytics spans the full scope and range of activity in higher education, affecting administration, research, teaching and learning, and support resources. The college/university thus must become a more intentional, intelligent organization, with data, evidence, and analytics playing the central role in this transition."¹²

The emergence of analytics is the result of the evolution of enterprise and instructional systems,¹³ which began in the 1990s when administrative systems were stand-alone, legacy systems. Hardware decisions pit mainframes against minicomputers. Business operations and information were siloed. Pre-1995 teaching and learning systems were fragmented. By the late 1990s, enterprise systems were becoming better integrated, resulting in data being more easily integrated. Over time, technology advances enabled more "fully integrated systems"¹⁴ allowing for greater transactional efficiency, information integration, reporting and business analytics, business intelligence, as well as recruiting and retention improvements.

Academic systems were slower to develop, but eventually course information systems (CIS), course management systems (CMS), and learning management systems (LMS) were developed. Later, Internet and web-based products and services began to emerge. This allowed for the convergence of administrative and academic systems at the enterprise level. Institutions began to address enterprise-wide systems including LMS and student information systems. Enterprise portals continued the evolution with the ability to access and integrate ERPs, LMS, and knowledge assets, creating a self-service foundation for students, faculty, and staff.

By 2004, the technology infrastructure fused networking, integrated software (e.g., ERP), security, and vast digital resources. Open source application software grew steadily, with technology focusing on security, open sources, web services, and network services.

For the past fifteen years, the emphasis of IT has been on the development of technology infrastructure. Today the emphasis goes beyond the infrastructure and includes business processes and strategic alignments. Calls for enhanced performance and demonstrated value have moved to the development of open-source/open-architecture developments and the ability to leverage the stack and the cloud. Today we are seeing the first generation of student/institutional portfolios, executive dashboards, and assessment/performance management systems within an open architecture environment.¹⁵ In addition, in the LMS 2.0, consolidations of campus LMS options were available. Database infrastructures including data warehouses began to be developed.

Analytics will be an essential future part of higher education. Institutions' previous efforts of capturing data, providing availability in data warehouses, and initial data mining efforts are foundational to the next generation of activities. Higher education is benefiting from the extensive business intelligence efforts found in the corporate world and will develop new integrated solutions within the learning environment as one takes advantage of the LMS, SIS, and other emerging tools.

Building Analytics Capacity

Academic analytics relies on the extraction of data from one or more systems, such as the CMS or a student information system. The data, which may be stored in a data warehouse for ongoing use, is analyzed using statistical software, and a mathematical model is generated. Based on the model and predetermined values, a particular action may be triggered, such as sending the student an electronic notification or initiating a personal intervention by college/university staff.

For example, data extracted from a student information system provides baseline student demographic, academic performance, and aptitude information. The CMS provides a snapshot of the student's course-related efforts by providing real-time interaction information that allows for comparison with peers. The two sources of data are combined to predict the probability of student success. Using this probability, the institution can decide whether to take certain actions such as inviting a student to a help session via e-mail or calling a student with an invitation to meet with an advisor.

Beyond the data, technology, and statistical requirements, academic analytics projects require skill and leadership. Three characteristics of successful academic analytics projects include:

• leaders who are committed to evidence-based decision making
• staff who are skilled at data analysis
• a flexible technology platform that is available to collect, mine, and analyze data.

Any academic analytics effort begins with leaders who are committed to decision making based on institutional data. Analytics can be used to examine key institutional issues, such as enrollment or retention, which by their nature are complex and often sensitive, but the decision to move forward with analytics depends on knowledgeable champions among senior administrators.

The second critical component to building an academic analytics initiative is staffing. Staff members involved in analytics efforts often include database administrators, institutional researchers, educational researchers, programmers, and domain specialists (e.g., student services, retention, development/advancement). Academic computing staff may be needed to collect information from various academic systems such as the CMS. The team must have the skill to build predictive models based on institutional data guided by educational research. Other staff may be needed to focus on policy development and clarify who has access to the data, how the data can be used, and which data-security models are required.

Since analytics requires data analysis, institutions will need to invest in effective training to produce skilled analytics staff. Obtaining or developing skilled staff may present the largest barrier and the greatest cost to any academic analytics initiative. Whether such staff are added to existing institutional research units or are cultivated in the IT organization, student-affairs divisions, or academic units will depend on the organizational culture and the locus of resources.

The third element in any academic analytics project is technology. A data warehouse is the key component of the technology infrastructure, housing information from a variety of sources in a common structure that enables data analysis. To populate the data warehouse, the institution will need to build a "bridge" between the application and the warehouse. For some applications, standard interfaces facilitate the transfer of data. For other applications, interface development requires significant programming effort.

Piecing together a coherent academic analytics effort can be difficult, requiring support from many units: enrollment management, institutional research, IT, the registrar's office, academic divisions, student affairs, and more. Standards must be agreed upon for the data (e.g., is enrollment based on head count on day seven after the start of the semester or on day ten?). Extracting information from academic systems requires careful analysis and programming effort. Building the appropriate models requires staff with statistics and educational research backgrounds. Creating interventions requires domain knowledge (e.g., advising, retention) and advising/counseling skills. For institutions to be successful in academic analytics projects, IT leaders must build a coalition of people.

Leading Change

The role that analytics can play within the learning environment will largely depend on the institution's vision of the next-generation learning environment. Part of that vision can be "actionable intelligence" where tools and data reduce risk of student failure and maximize the odds of student success.

The initial wave of learning analytics tools are emerging. They seek to improve the understanding of the ways students, faculty, and advisors can improve student retention and success. Much as various course management systems emerged in the 1990s, higher education is seeing the first generation of predictive modeling, adaptive learning tools, early warning tools, and new data visualization tools to enable decision makers to access and use data in a timely manner. As with early course management systems, institutions should also anticipate a significant evolution in tools and capabilities.

Leaders need to create an institutional culture to use analytics tools to maximize the potential for improved student access, student learning, progression, and success.

An institution should consider several key steps to the adoption of analytics:¹⁶

• Identify thought leaders for using data to solve instructional challenges.
• Build the existing predictive modeling capacity and expand across the programs.
• Identify what data are important and the metrics used to measure them.
• Identify best-in-class analytical and predictive modeling tools, applications, and processes.
• Embed changes in analytics in institutional processes.
• Aggressively develop organizational capacity for using analytics.
• Create a communication plan.

Identify thought leaders for using data to solve instructional challenges: As previously indicated, the three characteristics for a successful academic analytics effort are leaders, staff, and technology. It is the people at the institutions that are the most critical component to leading a successful change. One should begin by identifying individuals who are looking to make data-based decisions, which might include a mix of faculty, advisors, student services, and technology staff. Identifying a mix of people that includes those who are already making small decisions based on data with those looking to address larger institutional goals will be essential to success.

Build the existing predictive modeling capacity and expand across the programs: What programs and departments are already utilizing predictive modeling at institutions? Take a proactive approach and open the discussions with the campus community. A number of smaller models might already exist. For example, a department might have already connected attendance at orientation sessions for returning adult students with likely success in the programs. Institutions can also begin by examining their admissions process—what are the key factors to being admitted to the institution? Based on the existing models and those perceived models, how might they be combined to provide a more holistic view?

Identify what data are important and the metrics used to measure them and the alignment to institutional goals: Based on the key institutional goals, what data would help inform the potential solutions? For example, if the institution is focused on retention, what data might help inform staff on the potential for an individual student to remain at the institution? One might consider academic preparation (application data, placement tests, etc.), effort (learning management system, attendance, etc.), integration into the campus community (participation in learning communities, student activities, etc.), and willingness to seek help (visiting advisors, help centers, etc.). Each of the data sources could provide insight to the overall problem. The key element for success is identifying a starting point and continually adding new data to develop additional insights.

Identify best-in-class analytical and predictive modeling tools, applications, and processes: Many tools and resources are available to better serve students across their educational pathway. The field is seeing the increase in adaptive learning tools, early warning tools, use of social data to better understand student engagement and integration to campus, and new data visualization tools to enable decision makers to access and use data in a timely manner. What tools are institutions most interested in learning more about to improve student success? Leaders need to better understand how to select and use analytics tools for changing the learning equation to improve student access, progression, and success. They need to draw on data-supported evidence, which is now even more powerful with the expanded capabilities of learning analytics, predictive modeling, and tools that map to interventions that assist students. Leaders also need to understand the importance of building learning environments that support evidence and inquiry across the institution. Analytical packages should be considered as the first generation; be prepared to actively engage providers/consortia on new functionalities or even to migrate to new solutions.

Embed changes in analytics in institutional processes: Institutions should consider how to embed small analytics projects within existing programs, leveraging existing data and integrating into current student-success efforts. For example, if the biology department is seeking to increase retention of majors, what data might indicate a likelihood of students staying within the major? Leaders need to consider methods of going beyond reporting and find new ways to proactively assist students. In addition to a programmatic approach, the institution should identify mechanisms in which data is available for all areas that it can utilize.

Aggressively develop organizational capacity for using analytics: Analytics is a new tool for the next generation of learning. As such, the skills must be developed. While the need for the technical skills of data mining and statistical analysis is obvious, such skills as process analysis, assessment, and instructional design are also essential to the process. The goal for analytics must remain "actionable intelligence," and as such, the capacity for analytics must go beyond data and statistics and focus on how the information must be utilized.

Create a communication plan: In order to sustain and scale the development of an analytics agenda, it is imperative that leaders regularly communicate with stakeholders about the process and the outcomes. Focus and attention on how the analytics strategies affect performance, productivity, and value will form the foundation for the next-generation learning model. The organization use visualization tools to maximize the message while customizing reports for the right stakeholders. Regular communication will increase the trust and overall use of the analytics, which can build the culture of evidence and inquiry required to sustain the efforts.

The Future

Innovation has been characterized as new creative products, ideas, activities, or interventions that produce an improved result. The future of analytics promises to be both a sustaining and disruptive innovation for education. Analytics as a sustaining innovation refers to the normal upgrading and integration of analytics into current teaching and learning tools. Today, institutions can implement a variety of analytics solutions as part of the course management and student information systems. Analytics as a sustaining innovation will serve higher education by providing incremental improvements in the existing system, while not widely disrupting the institutional processes. An example of a sustaining innovation is using predictive analytics to identify at-risk students early so institutions can intervene in a timely manner to increase the likelihood of success. Research in the report Time is the Enemy by Complete College America references several ways to improve success for full-time and part-time students, including simplifying the registration process; accelerating the time to degree completion; blocking schedules with fixed, predictable classroom meeting times; forming peer support and learning networks; embedding remediation into the regular curriculum; and reducing time in the classroom through the use of online technologies.¹⁷

Analytics as a disruptive innovation refers to new products, ideas, activities, or interventions that require changing behavior/processes or modifying other products/services. Analytics in this form breaks with current practice to serve the student, faculty, and administrative users in radically different ways; it serves new populations (or serves an existing population in radically different ways) and, in so doing, creates entirely new systems to accomplish this.¹⁸ As the organization maps analytics strategies, both sustaining and disruptive innovations are possible

We can anticipate several new disruptive innovations from analytics:

• Utilizing "social" data to better understand student integration into campus. Research has found that environmental factors are equally as important as academic factors in student retention. How a student integrates into the social fabric, the formation of friendships and support groups, the adjustment into student housing, and similar factors all play an important role in student success. As the use of social media continues to increase, one could imagine mapping social connections to determine which students are having difficulty with connecting to the institution. Collecting, analyzing, and acting upon such data could potentially bring new groups together, ranging from housing, advising, and student groups.
• The growth of CRM as a collection point. Traditionally the "customer relationship management" (CRM) system has been focused on the admissions process. One could imagine future analytical tools coming together in a "learning relationship management" (LRM) system that would be open to faculty and advisors. The system would not only provide a central point for analytics data, but would also provide a way of tracking interventions and related results. The LRM system would provide a comprehensive foundation for end-to-end student support.
• Emergence of adaptive learning. If efforts to use analytics to predict success proved fruitful, the next significant step would be to use analytics to power adaptive systems that adapt to the learner's needs based on behaviors of the individual as well as of past students' patterns.
• Disaggregation of the data sources and the emergence of new analytics techniques. Analytics has focused primarily on integrating techniques into the course management and student information systems. When data from many different sources can be integrated, including audience response systems, publisher content, social media, and other data, new innovations will be possible.
• Mapping to interventions. Analytics can link suggested interventions to the use and impact of the interventions. If the intervention suggested utilizing the "math help desk," did the student use the resource? If so, for how long and while doing what activities? To enable such mapping, new systems must be established to share data between organizations to ensure privacy, while still allowing for impact.

Conclusion

If educational completion is one of the most important achievements for every American student, we need to leverage the technologies and analytical tools that will eradicate the most common educational mistakes (taking wrong turns, running out of academic gas, miscalculating the distance, underestimating the costs, and not having a "norm" to compare a personalized educational journey against). What might a futuristic analytics tool set look like? One that was personalized, adaptive to the individual learner needs, and that provided pathways and routes to maximize student success.

The world has become accustomed to using the Global Positioning System (GPS); now it needs an Educational Positioning System (EPS). All students would be furnished with this EPS, enabling them to navigate their educational journey in the same relatively simple manner they used their GPS system for locating their campus for the first time. Technologies currently exist that would allow students to map their educational starting point and destiny, determine how many educational units per dollar they are getting with their funding and how much time is left on their educational journey, interpret the academic gas tank indicators, and compare how they are stacking up against the educational norm during all points of their journey.¹⁹

Leaders can use these next generation of game-changing tools to develop actionable strategies and interventions to optimize institutional and student performance. And with these tools, institutions can focus on learner relationships, customization, and personalization and on interventions that meet learners where they are and help them get where they need to go. We need to embrace changes that optimize lifelong learning.

"The best way to predict the future is to invent it."²⁰ Higher education has both a great responsibility and a great opportunity to improve student success. Today, the demand for better metrics and improved productivity, accountability, and performance has brought an important tool to higher education in the form of analytics. The future holds much promise.

Notes

1. American Graduation Initiative (American Association of Community Colleges), http://www.aacc.nche.edu/Advocacy/aginitiative/Pages/default.aspx.
2. July 14, 2009, http://www.whitehouse.gov/the_press_office/Excerpts-of-the-Presidents-remarks-in-Warren-Michigan-and-fact-sheet-on-the-American-Graduation-Initiative/.
3. The Alliance of States, http://www.completecollege.org/alliance_of_states.
4. Complete College America, http://www.completecollege.org.
5. Sandy Baum and Kathleen Payea, Education Pays: The Benefits of Higher Education for Individuals and Society 2004 (Washington, DC: College Board, 2005).
6. U.S. Census Bureau, The Big Payoff: Educational Attainment and Synthetic Estimates of Work-Life Earnings (report, July 2002).
7. Robert Longley, "Lifetime Earnings Soar with Education," About.com, August 24, 2011, http://usgovinfo.about.com/od/moneymatters/a/edandearnings.htm.
8. Complete College America, Time Is the Enemy (report, September 2011), http://www.completecollege.org/docs/Time_Is_the_Enemy_Summary.pdf.
9. Alison Bailey, Tyce Henry, Lane McBride, and J. Puckett, Unleashing the Potential of Technology in Education (The Boston Consulting Group, 2011).
10. John Campbell, Peter DeBlois, and Diana Oblinger, "Academic Analytics: A New Tool for a New Era," EDUCAUSE Review 42, no. 4 (July/August 2007): 40–57.
11. Donald Norris, Linda Baer, Joan Leonard, Louis Pugliese, and Paul Lefrere, "Action Analytics: Measuring an Improving Performance That Matters in Higher Education," EDUCAUSE Review (January/February 2008): 43–67.
12. Phil Long and George Siemens, "Penetrating the Fog: Analytics in Learning and Education," EDUCAUSE Review (September/October 2011): 31–40.
13. Donald Norris, "Technology, Analytics and Lifting Out of Recession" (unpublished paper, July 2009).
14. Ibid., 9–13.
15. Ibid.
16. Adapted from Donald Norris, "Creating a Culture of Performance Measurement and Improvement" (presentation at the iStrategy User Conference, September 24, 2008), and Making the Numbers Add Up: A Guide for Using Data in College Access and Success Programs (report, Lumina Foundation, October 2009).
17. Complete College America, Time Is the Enemy.
18. Adapted from Geoffrey A. Moore, Crossing the Chasm: Marketing and Selling Disruptive Products to Mainstream Customers (New York: Collins Business, 2002).
19. Michael Mathews, "What If Steve Jobs Had Been an Educator?," from Bill Grave's The Learning Cloud blog, http://institutionalperformance.typepad.com.
20. Alan Kay, The Quotations Page, http://www.quotationspage.com/quote/1423.html.

Linda Baer has served in numerous executive-level positions in higher education including Senior Program Officer in Postsecondary Success for the Bill & Melinda Gates Foundation, Senior Vice Chancellor for Academic and Student Affairs in the Minnesota State College and University System, and Senior Vice President and Interim President at Bemidji State University. Currently, she works to inspire leaders to improve student success and transform institutions for the future. John Campbell is the Associate Vice President of Academic Technologies at Purdue University. During the past 10 years, Campbell has examined methods to use academic analytics to identify students at risk within courses. He was the founder of the Signals project, which has been featured on NBC and in the Chronicle of Higher Education.

This chapter is licensed under a Creative Commons Attribution 3.0 Unported License

© 2012 Linda Baer and John Campbell