Scenario

Dr. Harrier receives an e-mail from the teaching center asking how he feels about replacing the university’s existing LMS. In the geography courses he teaches in the classroom and online, he uses a complex map application that is integrated into the LMS, as are other features he uses such as discussion boards, wikis, and interactive testing. He doesn’t look forward to learning a new system, but he has been frustrated by the current system’s slow posting of updates. He checks the box that indicates he is open to the idea of a thorough assessment of the current LMS. Another question appears, inquiring if he would like to be part of the investigative task force for a new LMS. He decides he would.

During the first meeting of the task force, discussion centers on integration with Web 2.0 and cloud applications as a high priority. The team agrees their starting point will be to ask faculty members to identify their needs and desires of an LMS. Meanwhile, the student representatives on the task force will create a questionnaire to determine student preferences. Using this input, the task force creates a checklist of important features and assigns a point scale for each element. They will use this rubric to rate the LMS candidates as each is demonstrated by vendors or by the Learning Technologies Center.

Several hands-on demonstrations of each product are scheduled for the faculty, and Dr. Harrier tries to be present at more than one session for each product. When the faculty responses favor two LMS options, he volunteers to teach an online course using one, a commercial product similar to the LMS the university already uses. He works closely with Dr. Perkins, who is teaching a hybrid course in history using the other LMS candidate, an open-source option. He and Dr. Perkins compare notes on their experiences each week, and following the tests, they submit their findings. When these are aggregated with the findings of the other instructors participating in the pilot courses, the task force can see that the open-source LMS will best meet the needs of the university.

Over the next two years, Dr. Harrier will work with faculty members during transition, confident that he and the task force have chosen the best product for the school.

1. What is it?

For most institutions, the LMS is the most significant enterprise application for teaching and learning. It serves as the resource hub for the institution’s classes and delivers courseware to students. Depending on the particular system and its implementation, an LMS might also manage course registration, course scheduling, discussion forums, blog sites, student scores, and student transcripts. At many institutions, the current LMS is five or more years old, and although the application has likely been upgraded several times, the teaching and learning context might have changed considerably in that time. Moreover, other products, including open-source options, have emerged. All of this raises the question of whether the institution should conduct a complete evaluation of its LMS strategy. Formerly, such an evaluation would generally have been the province of the IT department. Today, portions of the analysis are handled by members of the academic community, including faculty members, students, administrative personnel, library staff, and teaching and learning center staff, all of whom play a growing role in the present wave of LMS evaluations.

2. How does it work?

When a school has decided to evaluate an existing LMS, the project is typically led by a task force that includes some combination of IT specialists, faculty members, librarians, instructional designers and technologists, administrative leadership, staff, and students. This group identifies stakeholders and establishes a communications plan. One common approach is to provide a regularly updated website where the schedule, methodology, and reports can be monitored by all interested parties as the task force undertakes fact-finding and analysis.

To evaluate the LMS candidates, the task force constructs tools that may take the form of rubrics, matrices of desirable features, or rosters of strategic and tactical priorities. Stakeholders are queried about their interests and needs, and test classes may be set up to run new LMS choices on an experimental basis. Vendors might be asked to send information about their products and possibly to stage demonstrations of them. Once results are in from these activities and the results have been analyzed, the task force can present its findings to the institution’s decision makers. The entire process, from initial meeting through completed transition to a new LMS, can take up to three years.

3. Who’s doing it?

Like several other institutions that completed an LMS evaluation, the University of North Carolina at Charlotte has made its process available on the web. Postings include descriptions of the procedures and the rubric it used to evaluate available LMS options to find the right one for the institution. Faculty member input reflected a strong preference for an open-source product, due to its ease of use and sufficiency of available pedagogical tools. Further, the open-source model gave the IT department control of the development path of the LMS; if they wanted to add a new feature or remove an old one, they could do it locally without depending on a vendor’s timetable. As a result, UNC Charlotte elected to replace its existing commercial application with an open-source system. In addition, the team identified cost savings that would accrue to the university once the new LMS had been implemented.

At the San Diego Community College District (SDCCD), the LMS review was guided by the Online Pathways Unit, which serves all nine of the district campuses and oversees roughly 500 online and 900 web-enhanced courses that are hosted each semester. The group began with a survey of faculty member needs. Then, after setting up and analyzing faculty member “test-drives” of numerous products, they used the results to narrow the field of LMS candidates. The finalists were evaluated using a decision matrix that prompted stakeholders to specify their LMS preference based on such factors as vendor reliability, required features, efficiency of migration from the existing system, and support services. As a result of scores collected from the matrix, SDCCD chose a new commercial LMS by the same vendor that had provided its existing one.

4. Why is it significant?

An existing LMS tends to take on some measure of inertia: Faculty members are inclined to perceive it to be the only option, with only pioneering instructors venturing into alternatives that better meet their needs. The LMS serves as the linchpin of an institution’s teaching and learning enterprise, though, and as such it should provide maximum value and flexibility for the faculty. On campuses where the majority of the faculty have been cautious adopters of LMS features beyond the posting of a syllabus or the use of the grade-book feature, an effective evaluation process can spark interest in a broader selection of tools, such as discussion boards, e-portfolios, wikis, and online quizzing. Regular evaluation can alert the faculty and other stakeholders to the options available and raise awareness that these large and expensive applications can deliver interesting and effective teaching and learning tools when the right product is selected.

5. What are the downsides?

Not all faculty members embrace the notion of an LMS evaluation. If a decision is made to replace the LMS, such an undertaking can bring with it a steep learning curve, something that may unsettle users who foresee a need to retrain. Administrative applications, like the student information and authentication systems, might have to be reintegrated, as might numerous ancillary applications such as lecture capture or clickers. The evaluation itself can be costly, and if an upgraded or different system is chosen, bridge funding during the transition can be a considerable expense if the institution hosts and pays for both the outgoing and new systems for a time. The transition phase can also be disruptive to the instructional process, as corrections and adjustments are often necessary for newly converted courses.

6. Where is it going?

As changes in education delivery and redefinition of student needs continue to reshape the modern campus, colleges and universities might find themselves adapting to pedagogical changes more frequently than in the past. This, in turn, might result in more regular evaluations of an LMS. Subsequent evaluation cycles might be smoother, though, if many of the participants come to the table with prior experience with the process and better informed and prepared to deal with the issues that may be raised. In cases where an institution has had to maintain (and pay for) two LMSs during a migration from one system to another, subsequent transitions might be kept shorter, saving money and eliminating some complexity.

7. What are the implications for teaching and learning?

Because the LMS is the central technology component of teaching and learning, a review of it is more faculty-driven than that of other campus-wide systems. The examination of the LMS forces an institution to take a hard look at its teaching practices, to educate the faculty that the LMS can be more than a “course website,” and to invite them to use the full spectrum of teaching tools it provides. In so doing, it keeps the instructional environment fresh and spurs competition in the marketplace. Many colleges and universities have made their LMS evaluations public, posting rubrics, reports, and data freely online. This has allowed those undertaking a similar evaluation to use the institutional experience of their peers to improve their own processes, enabling the kind of collaboration that reinforces collegial relationships within the academic community. Regardless of the triggering mechanism for an LMS evaluation—whether the institution is experiencing administrative changes or an application contract is up for renewal—the process inspires a reexamination of the instructional goals the system is designed to support. At the same time, it brings careful scrutiny to the features currently employed and those that may be used more extensively in the future.

May 2011

© 2011 EDUCAUSE
This work is licensed under a Creative Commons
Attribution-NonCommercial-NoDerivs 3.0 License.
http://creativecommons.org/licenses/by-nc-nd/3.0/

educause.edu/eli

EDUCAUSE 7 Things You Should Know About™…

EDUCAUSE is a nonprofit membership association created to support those who lead, manage, and use information technology to benefit higher education. A comprehensive range of resources and activities are available to all EDUCAUSE members. For more information about EDUCAUSE, including membership, please contact us at info@educause.edu or visit educause.edu.