Institutional Self-Assessment

EDUCAUSE LEARNING INITIATIVE | AUGUST 2015

Scenario

Dr. Riley is director of the technology and engineering library on the campus of a large state university. As part of an institution-wide funding cycle, he is preparing an assessment to compare library IT services with those at peer institutions. The assessment includes a survey of faculty, staff, and students who use this library, as well as data from an independent research group that looks at public universities across the nation. The survey instrument, which was used two years ago, contains questions about how important IT services are to patrons and how satisfied they are that existing services meet their academic needs. This year, the survey also includes a section that invites respondents to identify IT services they would like to see offered or improved at the library.

When Riley and the survey team review the survey results and compare them to the data from two years ago, they see that, in general, satisfaction is up among library patrons in most areas. But it has dropped sharply in one respect. Students are dissatisfied with the wait times for access to 3D printers.

The independent research data show that the library has 33–50% more 3D printers than comparable libraries, but students clearly want to see more—or perhaps faster—3D printers made available. At the next budget meeting, Riley will cite those trends along with data that show the engineering program has grown 20% more than similar programs in nearby states. He will argue that funding high-speed 3D printers for the library could help keep the university at the cutting edge of high-tech offerings in the region.

The survey also offers insight into how Riley might expand the library’s hardware lending program. Many respondents want to be able to borrow drones from the library. Benchmarking data from other sources indicate that many institutions, both peers and non-peers, currently lend drones or are considering doing so. Working with professors in engineering, journalism, and broadcasting and leveraging data from some of their grant requests, Riley will soon shape a proposal for funds that would allow the library to lend drones.

1 What is it?

Institutions are increasingly being asked to demonstrate the value of IT investments. Similarly, it is important for an institution to understand its current state before embarking on a major change. An institutional IT self-assessment frequently provides justification for existing or proposed projects and establishes new directions. Such assessments can function the way trail markers do on a map at a trailhead. Hikers search for the “you are here” designation and select their preferred route. Subsequent markers along the selected path may specify distance travelled and how much of the trail remains to be completed, much as periodic reassessment allows campus decision makers to monitor progress of an IT initiative.

2 How does it work?

IT assessments vary according to the goals and the tools used. An assessment could focus on a single initiative or on a broad program of change. Those involved in the assessment must identify what will be measured and who will do the measuring. The choice of an assessment instrument is often driven by the direction and character of the assessment. For example, a department engaged in evaluating IT in an existing learning center might use an instrument to measure existing or proposed IT services against a baseline set of computing facilities and services in a learning space. Such an effort might use a rubric, such as a scorecard or an IT maturity index, or it could opt for a relative measure, comparing IT holdings at a learning center with those of peer institutions that offer a benchmark. A third alternative would be to research best practices and measure an institution’s current offerings against an ideal.

3 Who’s doing it?

A range of assessment options are available for higher education. For those looking to browse for assessment tools, the Inventory of Higher Education Assessment Instruments website from Stanford University presents a list of assessment instruments as a resource for state policy makers interested in pursuing assessments. A number of colleges and universities have chosen to participate in the Higher Education TechQual+ Project, which offers a set of tools that focus on IT outcomes to determine how effective technology services are at a given institution. An online survey tool helps these institutions monitor the performance of their IT suite and track what their campus community wants and expects from their services. The Measuring Information Service Outcomes (MISO) Survey, employed by more than two dozen colleges and universities, measures how students, faculty, and staff view the library and computing services. Data are gathered on which services are most important and how satisfied participants are with each. The EDUCAUSE Core Data Service (CDS) gathers institutional data on IT financials, staffing, and services, establishing benchmarks along several dimensions of IT that allow institutions to compare their status in various areas to that of other institutions. EDUCAUSE uses CDS data to build maturity and deployment indices for topics such as analytics or research computing. For those looking to analyze learning spaces, EDUCAUSE offers the Learning Space Rating System, which provides a way for an institution to self-assess classrooms with respect to their readiness to support active learning.

4 Why is it significant?

The rapidly changing landscape of IT in higher education obliges institutions to choose among competing educational priorities. IT self-assessment is increasingly valuable to help with those decisions. In part this is because decision makers tend to trust data to provide answers to difficult questions. Assessment data can uncover the value and effectiveness (or the lack of either) in IT initiatives, countering (or supporting) suggestions that a technology is being deployed simply because it is new or popular. Polling users about what they like or dislike in current services underscores the link between IT services and those who use them and helps ensure that changes will support the needs of the community. An IT self-assessment could lend credence to the direction or goal that is under consideration. It might support requests, claims, and hunches about where the college should go next, or it could identify potential risks, allowing a college or university to strengthen infrastructure and improve practices.

5 What are the downsides?

Institutional IT self-assessment is not suitable for all initiatives, and it is not without its costs. The process uses resources such as staff time and administrative financing, and results may not be as clear cut as participants would like. People may end up not using the data that emerge or deciding to revisit the data in future. Some campus groups may object to the selected rubric or benchmark, maintaining that it doesn’t accurately measure the institution’s readiness. IT assessments can generate false positives or negatives, and results could trigger new controversies. IT self-assessments may work against institutional agility. Committees can be overwhelmed by data or become so focused on collecting it and subjecting it to multiple analyses that they hesitate to act. Once an assessment is completed, there can be a feeling of responsibility to act upon the results to justify the costs of the process.

6 Where is it going?

As costs rise in higher education, pressure increases to provide accountability. Students and parents are eager to find ways to ensure they receive educational value for their money, and the data provided by self-assessment instruments can address that need. Reporting on data among institutions is becoming more common, suggesting consumers will come to expect it. Self-assessments and benchmarking exercises may fuse with analytics so that colleges and universities will be able to do real-time IT self-assessments, allowing for rapid decisions based on data. As IT assessment becomes more routine, individual faculty members may find they need to justify funding and will elect to subject their projects to self-assessments in an effort to clarify why their project is important, what it will accomplish, and what the expected result might be.

7 What are the implications for teaching and learning?

Institutional IT assessments that are used to weigh strategic options may have long-term benefits by bringing focus to the changes in teaching and learning. In part this is because assessments often furnish the kind of evidence that can persuade institutional leadership to provide financial or other types of support for a project. As a consequence, teaching and learning projects that yield clear data paths with understandable results may be given precedence over those that do not, while initiatives that offer important but less quantifiable teaching and learning benefits could fail to get the recognition or support they need. Effective IT assessments, however, will provide evidence not just about what works and what doesn’t but also about the conditions in which particular tools or services are more effective. Ideas, projects, or initiatives may function well in some situations but not others, and assessments that recognize that distinction will contribute to the drive toward targeted teaching and learning.

educause.edu/ELI | ELI 7 Things You Should Know About ... ™

© 2015 EDUCAUSE. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 4.0 License.

EDUCAUSE is a nonprofit membership association created to support those who lead, manage, and use information technology to benefit higher education. The EDUCAUSE Learning Initiative is an EDUCAUSE community committed to the advancement of learning through the innovative application of technology. For more information about ELI, including membership, please contact us at info@educause.edu.