Assessing Reference Service Quality: A Chat Transcript Analysis

Abstract

In 2016, members of the University of Kansas Libraries’ Reference Services unit participated in a peer review of 60 chat transcripts from the Fall 2015 and 2016 semesters. This project grew out of the need to begin assessing the level of reference service quality provided at the University of Kansas (KU) Libraries.

In-person and virtual reference assistance is provided at the two main libraries up to 12 hours per day, six days per week, when classes are in session. Two main libraries plus four smaller branches make up the KU Libraries system. Staffing of reference services at the KU Libraries has evolved over time, from a mix of faculty-equivalent librarians and library staff to the current staffing model, which employs three full-time staff members and two half-time staff members, complemented by graduate student employees. While the current desk staff are all well qualified, very few have degrees in library science. Therefore, extra care must be taken to train reference staff on how to conduct a proper reference interview, and additional time is required to monitor reference stats to ensure that questions are being answered properly.

Previous local studies have been conducted in this vein. In 2007, librarians rated 2,300 chat transcripts from a two-year period to determine whether teaching moments were being taken advantage of, using ACRL’s Information Literacy Standards for Higher Education as criteria. A follow-up study took place in 2008, examining 50 of the previous 2,300 transcripts in detail to identify opportunities and best practices for incorporating instruction in virtual reference services.

Since 2008, KU Libraries have not conducted additional formal studies to examine the quality of reference services provided. During the Fall 2015 semester, chat questions made up approximately 43 percent of overall reference questions asked at KU Libraries reference desks. This increased to 47 percent during the Fall 2016 semester. As chat becomes an increasingly common venue for reference questions at KU, this transcript analysis is a first step toward measuring the quality of reference service quality. Chat transcripts are attractive as a starting point since they provide a comprehensive snapshot of an interaction with a patron, and can be examined in a variety of ways to inform best practices in virtual reference. This paper describes the study’s findings, and presents the project methodology alongside referenced literature as a resource for other librarians who may be looking for a starting point in establishing broader evaluation of reference services in general.