Caliper

EDUCAUSE LEARNING INITIATIVE | MARCH 2016

Scenario

Three years into a learning analytics initiative, officials at State University have seen positive results in course grades and student retention. The implementation of various Caliper-compliant applications, including an LMS replacement a year ago, promises even deeper insight into the connections between learning activities and outcomes. Caliper is a framework for learning data that enables the capture of highly granular data consistently across learning applications. Steve Holton, the university’s CIO, has been working with the provost to develop tools and services that capitalize on Caliper data. For example, through Caliper instrumentation, a quizzing tool for math gathers data such as when students take quizzes, how long they spend on each problem, what sequence they follow, and which resources they consult. Crunching data about these events helps designers associate learning tactics with outcomes, which informs the structure of future quizzes and exams. A similar project, an online protein-modeling application, produces Caliper data that can indicate the effectiveness of learning activities based on how students interact with them.

Since the implementation of the new LMS, the IT staff have deployed several Caliper-compliant online learning tools, and data from these applications can be easily combined in the LMS. The university encourages faculty and instructional designers to adopt only Caliper-compliant tools. Holton believes that the ability for IT and educational staff to work easily with data from a range of applications will not only provide gains in measurement and assessment of learning activities but will also lead to more sophisticated early alerts and better interventions, a more personalized curriculum, and more-comprehensive student records.

With a full year’s data to work with, Holton and his staff are prepared to unveil the first set of analytics tools and improvements that use Caliper information. In development is a combination of homegrown and vendor services that feed learning dashboards, both for faculty and students, as well as applications that tie into the university’s advising system. With more and better data at their disposal from an integrated platform of tools, advisors should be able to provide better guidance, with more specific recommendations, earlier in the term.

1 What is it?

Caliper is a standard that enables the collection, storage, and transport of data about learning. As a learning analytics framework, Caliper provides a common-gauge rail for disparate applications to use and share data from student interactions with learning software and administrative systems. Increasingly, a student’s learning environment extends beyond the campus LMS to other platforms and applications—often integrated using the Learning Tools Interoperability (LTI) specification—that generate data in a range of formats, often housed in disparate databases. Caliper is designed to overcome this disconnectedness by enabling data capture from student online interactions, devising a universal vocabulary to describe these events, and presenting collated data for use in learning analytics, research, and other learning-related endeavors. In much the same way that HTML codified an open standard for transfer and display of information, Caliper offers an open standard for learning-event data.

2 How does it work?

Caliper provides an information model that describes concepts, entities, and relationships. It is extensible, built on what Caliper calls “metric profiles” or “learning profiles” that describe general learning activities, such as enrolling in a course or taking an assessment. Developers can add profiles to accommodate innovation and can extend profiles by adding vocabulary to describe undefined activities. Data that conform to Caliper can be put into a common format for distribution and used across multiple learning environments. In its current release, Caliper establishes a format for collecting data from multiple sources and placing those data in a Learning Record Store (LRS). Various learning applications can access these data for learning analytics or other programs.

A typical student interaction might begin with notice that a student has begun an online assessment. With Caliper, that event can be connected to other data, such as a timestamp, the course name, or the name of the campus where the test is being taken. After the student completes the assessment, the learning service or application could generate an outcome file, also based on the Caliper framework. The data could be made available to the original application running at the student’s campus or to applications at other colleges and universities.

3 Who’s doing it?

The IMS Global Learning Consortium sponsors Caliper, and several institutions are involved. The University of Michigan, for example, is leading the building of the Caliper specifications. Developers there are working with faculty to build an in-house lecture-capture system that will generate media profiles that enable data capture by Caliper. In the physics department, a team is working on Problem Set Roulette, an online drill-and-practice program that uses data originally held in a databank of previous exams and questions. Developers at the University of California, Berkeley, are completing three Caliper-related projects: a set of interrelated LTI tools built in-house, including a media gallery and real-time whiteboard, which have been instrumented to emit Caliper events; a shareable LRS that ingests Caliper data and provides APIs for student privacy controls and anonymized access to researchers; and a project pilot that presents LMS Caliper event data in student-facing activity dashboards. At the University of Texas, teams are building an analytics workbench for faculty and students to analyze Caliper-managed data originally captured from the Canvas LMS. The analytics from the workbench project can be used to structure course reports that can be scheduled at a frequency set by faculty.

4 Why is it significant?

Caliper provides data structure and semantic interoperability, resulting in an improved exchange of information across applications and institutions. Because Caliper defines a common language for labeling learning data, it could set the stage for an ecosystem of higher-order learning analytics. As an information service, Caliper holds considerable potential for powering educational research, delivering information about machine learning, learner profiles, and predictive analytics algorithms. At the same time, the Caliper service ensures that as innovation continues, higher education can collect and collate data to provide a common set of data from students as they interact with applications and each other.

5 What are the downsides?

Because applications and services that use Caliper data sit outside Caliper’s purview, the long-term success of Caliper depends, in part, on developers making their products Caliper compliant. It also requires faculty and others to adopt Caliper-compliant tools. Also, if the use of Caliper extension capabilities is not consistent across learning tools, data from those applications will not be fully interoperable. In campus use, Caliper presents administrators with the usual privacy and security issues associated with any service that handles student data. In addition, institutional review boards and other administrators might question the extent to which student interactions in learning are being measured and whose eyes are on these measurements. The Caliper model raises questions about data ownership, whose data are being gathered, whether campus administrators are merely stewards of those data, whether the data could become part of a student’s extended transcript, and whether students can take their data with them when they leave college.

6 Where is it going?

Because Caliper allows third-party services and applications to access vast quantities of rich, organized data, the service will enable—and perhaps supercharge—new development. Expect applications and services to emerge that use Caliper data and return those data to learners, advisors, instructors, and students in the form of dashboards, reports, and alerts. The volume of data collated through Caliper could increase the accuracy of predictive models. As more profiles are created, the increasing data volume and complexity may strain research methods, and the challenge of harnessing the Caliper data stream may result in new research avenues and approaches. As Caliper itself changes through upcoming releases, the current set of profiles and definitions will be enhanced. Going forward, Caliper will create new ways to receive data from applications and services and to generate improved documentation.

7 What are the implications for teaching and learning?

While not an end-user product, Caliper could change the experience for students, faculty, and advisors. For students, education is increasingly about engagement with numerous applications on a variety of hardware platforms, such as mobile devices, info-gathering drones, smart watches, or even game consoles. Data from these and other sources could add valuable information to a student’s learning archive, but in the current environment the data live in separate silos and are costly to gather and collate. Caliper could change the learning landscape by enabling the collection of data from a variety of platforms and making them available for analytics and other uses. For faculty, Caliper provides a data-exchange mechanism for building high-value analytics services that enable instructors to compare the learning patterns of students within a course, across the campus, or elsewhere in the world. Finally, Caliper can furnish more and better data to advisors, who will increasingly rely on integrated systems to help guide students through course selection and degree progress.

educause.edu/ELI | ELI 7 Things You Should Know About ... ™

© 2016 EDUCAUSE. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 4.0 License.

EDUCAUSE is a nonprofit membership association created to support those who lead, manage, and use information technology to benefit higher education. The EDUCAUSE Learning Initiative is an EDUCAUSE community committed to the advancement of learning through the innovative application of technology. For more information about ELI, including membership, please contact us at info@educause.edu.