Usability test results

This text summarizes the results of the Libraries' recent usability test against its website.

Contents

1. Demographics
2. Tasks
3. Results
4. Comments
5. Analysis
6. Recommendations

Demographics

Fourteen (14) usability tests were conducted by the folks in DAIAD with the help of Sherri Jones, Aaron Bales, and Scott Van Jacobs December 1 - 3, 2004. Most usability experts think it is only necessary to get the input of six or so people.

All of the test takers were students. About half of them were graduate students, and half of them were undergraduate students. As expected, most of the students were studying in the humanities, and the least represented were students from the physical sciences.

When asked how often they used the website, the responses were evenly split between "a lot", "sometimes", "rarely", "never", and "I don't know."

Tasks

Participants where asked to accomplish the following tasks:

1. What is the call number of the book entitled French Paintings by Lorenz Eitner?
2. What is the name, email address, and telephone number of a librarian who can help you with a chemistry question?
3. The book entitled British Masters: A Survey and Guide by Horrace Shipp is currently checked out of the library. Request that the library have this book returned so you may borrow it.
4. Download and/or email to yourself the full text of three scholarly articles about AIDS from at least two different journals.
5. What are the titles and call numbers of three books about the American Revolutionary War?
6. The Libraries does not own the book entitled Folding the Universe by Peter Engel (published by Vintage Books in 1989). Request the Libraries borrow this book from another library on your behalf.
7. Does the Libraries own volume 45, number 4 of the journal Challenge, and if so, then what is its call number?
8. Download and/or email to yourself the article entitled Painting Thoughts, Listening to Images by Angela Dalle Vacche found in volume 46, number 4 (Summer 1993) of Film Quarterly.
9. What is the title of at least one book put on reserve by Dr. Kimbra Smith?
10. What is the URL of Dissertation Abstracts Online, a specific index of theses and dissertations?

Results

Most people were able to do most of the tasks. For example, the following tasks were accomplished by eleven or more of the participants:

• #1 - What is the call number of the book entitled French Paintings by Lorenz Eitner?
• #2 - What is the name, email address, and telephone number of a librarian who can help you with a chemistry question?
• #3 - The book entitled British Masters: A Survey and Guide by Horrace Shipp is currently checked out of the library. Request that the library have this book returned so you may borrow it.
• #6 - The Libraries does not own the book entitled Folding the Universe by Peter Engel (published by Vintage Books in 1989). Request the Libraries borrow this book from another library on your behalf.

The following tasks stumped about half of the people:

• #4 - Download and/or email to yourself the full text of three scholarly articles about AIDS from at least two different journals.
• #9 - What is the title of at least one book put on reserve by Dr. Kimbra Smith?
• #10 - What is the URL of Dissertation Abstracts Online, a specific index of theses and dissertations?

The most problematic tasks -- the tasks alluding most people -- were:

• #5 - What are the titles and call numbers of three books about the American Revolutionary War?
• #7 - Does the Libraries own volume 45, number 4 of the journal Challenge, and if so, then what is its call number?
• #8 - Download and/or email to yourself the article entitled Painting Thoughts, Listening to Images by Angela Dalle Vacche found in volume 46, number 4 (Summer 1993) of Film Quarterly.

Comments

• [Comments on advanced OPAC search and said,] "different" and "a lot of options" --Freshman
• Add more catalog searches to the ND Catalog Quick Search on home page. Please move course reserve OPAC link to top of Course Reserve page. --Grad student
• Change FindText so that it will search for books titles correctly. Website easier to use because things are "up front". --Grad student
• Couldn't find Grove Dictionary of Music. Move library links to top or make more prominent. --Senior
• Doesn't think basic search works well. Make it better. --Senior
• Don't need graphic--pseudo-popups. Looks good. Lots of information. Can always ask a librarian. --Sophomore
• Graphic buttons on main page too big. Theme like blue and gold. How to find reserve material? [He did know and wanted the answer.] --Undergrad
• Images too big. While "All Libraries" should be larger, because that information is important. --Grad student
• Implement simpler options. Like Google. --Sophomore
• Just use website for EBSCO. Or, "I will come to the library." Would be nice to talk to "freshman" librarian. I use EBSCO because I had training on it and it is easy to use. I would have a link to most frequent places visited. --Freshman
• Link directly to advanced catalog. Pretty good website. Easy to navigate. Things linked on main page. --Grad student
• More user-friendly help section. --Grad student
• Search films in audio. Couldn't find AIDS. Would be good to have journals in one place. --Junior
• When I first started using this, it through me, and i expected a drop down to be what type of thing I'm looking for. I didn't realize this was a link to the catalog. Though I had to enter something. You made WorldCat easier to find, which is good. it's gotten harder to find book reviews. Article First was good but WorldCat took away. --Grad student

Analysis

In many ways, the results of this usability study were not surprising. They seemly reflect the behaviors and expectations of many student users across the nation regarding library websites. Students, especially undergraduate students, have difficulty choosing most appropriate indexes (databases) for academic research, and their understanding of the databases' scope and features are rudimentary.

For example, task #8 (Download and/or email to yourself the article entitled Painting Thoughts, Listening to Images by Angela Dalle Vacche found in volume 46, number 4 (Summer 1993) of Film Quarterly) could have been accomplished by searching the catalog for Film Quarterly, searching the site for Film Quarterly, or searching E-Journal Locator for Film Quarterly. Instead, users searched the catalog for the article title or author name. Additional techniques included selecting a journal index and searching for the author or article title. The distinction between and where to find a journal title, a book title, and an article title seemed to be lost on many of the participants.

Task #5 (What are the titles and call numbers of three books about the American Revolutionary War?) stumped most people because they used the keyword search of the OPAC with words such as american and revolution. While this resulted in many hits, and users rattled off a number of call numbers, few of these items were on the American Revolutionary War. Even when users used the subject index of the catalog, no hits were returned.

Task #7 (Does the Libraries own volume 45, number 4 of the journal Challenge, and if so, then what is its call number?) alluded people because they were unable to understand the screen layout. Oftentimes people correctly searched the catalog for the serial Challenge, but the resulting screen confused them. There were too many titles named Challenge. Furthermore, when users browsed the resulting list more thoroughly, the display was too difficult to read. One person even highlighted the volume/number information and said, "The information is not here."

Finding articles on AIDS (Task #4) was the next most difficult task to accomplish. The problem here was not so much using the bibliographic databases but selecting them in the first place. For example, if users were given a topic in their subject area and then asked to retrieve a few articles the task was almost trivial. Users seem to know about a limited number of databases and stick to those databases almost exclusively. "I use EBSCO because I had training on it and it is easy to use."

All is not lost. Most people were able to do most of the tasks. Site-wide navigation did not seem to be a problem. People appreciated the number of choices on the home page and were able to easily navigate back to the home page and begin new tasks -- people did not get lost in the site. Finding the call number of a book was easily accomplished. Identifying the contact information of a specific librarian was easy. Requesting an interlibrary loan or book recall was simple for the majority of the people.

We as a library must define for ourselves a level of success. It is unrealistic to expect 100% of the people to do 100% of the tasks. At the same time, it is irresponsible to measure success as nobody accomplishing any of the tasks. What does success look like? Do 80% of the people need to be able to accomplish 80% of the tasks? This question is still unresolved.

Recommendations

Based on these studies there are a number of recommendations:

1. Conduct a number of surveys, usability studies, and focus group interviews on the catalog portion of the website before the implementation of the next version of the OPAC. Such actions will help the Libraries set priorities for the catalog's features and better insure the support of the students and faculty when the upgrade is complete.
2. Integrate customization and personalization features into the website. Users don't necessarily know what databases to use, but they do know about themselves. By enabling users to log in and identify themselves we, the Libraries, will be able to recommend starting points for library research through a combination of users' University department, field of study, and rank.
3. Re-phrase, re-work, and re-examine how the organization of bibliographic databases and journals are presented on the site. Implementing a step-wise approach to database selection in the form of a mini-reference interview may be one solution. Rewording headings to be more active may be another.
4. Actively market the website. The website is not too difficult to use, rather, people do not know what it can do. By more actively marketing the site more attention will be drawn to it, and we, the Libraries, will receive more feedback regarding its functionality.
5. Continue focus group interviews and usability studies, but in the future, do the usability studies in La Fortune where there is a larger user population.

Author: Eric Lease Morgan <emorgan@nd.edu>
Date created: 2004-01-05
Date updated: 2007-03-21
URL: http://www.library.nd.edu/daiad/morgan/usability-results/