Assessing Undergraduate Information Literacy Skills Using Project SAILS
J.B. Hill, Carol Macheak, and John Siegel, University of Arkansas at Little Rock

Abstract
The University of Arkansas at Little Rock (UALR) is a metropolitan institution that serves a large number of students from nontraditional, first-generation, and ethnically diverse populations in an increasingly online environment. One challenge for UALR is how the university can encourage student retention and graduation, as undergraduate student success is a major focus of current campus initiatives. Recognizing that information literacy skills contribute to student success, UALR conducted an assessment of undergraduates using the Project SAILS cohort test. Collecting data using a combination of the SAILS instrument, a Blackboard community, and WordPress forms, librarians at the Ottenheimer Library collected information literacy data for freshmen students in ten sections of lower-level courses and senior students in twenty-four sections of upper-level level courses. The results of the SAILS assessment indicate that students are arriving on campus without the research skills needed for academic success. Consequently, librarian and faculty partnerships are important to provide research intensive experiences throughout the curriculum to promote the development of research skills and student success. 

Keywords: academic libraries; assessment; information literacy; Project SAILS

Introduction
The University of Arkansas at Little Rock (UALR) is a metropolitan institution that serves an ethnically diverse population of undergraduates, including many nontraditional, first-generation, and transfer students. The average age of an UALR student is twenty-seven years old with approximately 47 percent of the student body attending classes on a part-time basis. Over 90 percent of the student population commutes to campus, and students are increasingly taking classes online.
Due to the nontraditional nature of the student body, students often take longer to complete their degrees and the university’s retention and graduation rates are lower than universities with traditional on-campus populations. Educational attainment in Arkansas is low by national standards, with only 18.9 percent of adults holding a college degree and 41.2 percent of college students successfully attaining a degree within six years. The state has established an ambitious goal of doubling the number of Arkansans with college degrees by 2025 and has begun basing university funding in part on student completion rates. As a result, UALR has established student success as a critical organizational objective and faces the challenge of providing educational experiences that promote student success. 
Students enter the university with a wide range of research skills and experiences. While the research tools, resources, and issues vary among disciplines, research/information literacy skills are important to completing assignments in all disciplines. Librarians have long held that the ability to find and process information is vital to a student’s personal, academic, and professional success. 
The Association of College and Research Libraries (ACRL) has established Information Literacy Competency Standards for Higher Education that define information literacy as “the set of skills needed to find, retrieve, analyze, and use information.”1 According to the ACRL competency standards, an information literate person should be able to: 
•	Determine the extent of information needed
•	Access the needed information effectively and efficiently
•	Evaluate information and its sources critically
•	Incorporate selected information into one’s knowledge base
•	Use information effectively to accomplish a specific purpose
•	Access and use information ethically and legally
•	Understand the economic, legal, and social issues surrounding the use of information.2

The importance of information literacy skills has been routinely noted by regional accreditation organizations. While the Middle States Commission has been the most detailed and explicit in promoting information literacy, others such as the New England Association of Schools and Colleges and the Western Association of Schools and Colleges include information literacy in their standards. The Southern Association of Colleges and Schools, the North Central Association of Colleges and Schools, and the Northwest Commission on Colleges and Universities do not use the phrase “information literacy,” but they do state the importance of library instruction.3
Recognizing that information literacy skills can contribute to student success, the authors decided to conduct an assessment of the research and information literacy skills of current freshmen and seniors using Project SAILS. A nationally-recognized instrument, SAILS allows universities to determine the level of students’ skill attainment and compare this to national norms of information literacy skills. 
Information Literacy Assessment and Project SAILS
A number of information literacy assessment instruments have been developed since 2000. Widely-used assessment tools administered to undergraduates at universities across the United States include Project SAILS, ETS iSkills, Madison Assessment Information Literacy Test, and the South Dakota Information Literacy Exam.4,5,6,7
Developed by Kent State University with assistance of grant funding from the Institute of Museum and Library Services (IMLS), Project SAILS provides a multiple-choice knowledge test targeting a variety of information literacy skills. As Blixrud indicates, the SAILS instrument was developed to assist universities in determining the level of student information literacy skills at the point of admission and graduation, with the intent to correlate the skills to student success and retention.8  The goal of SAILS was to be “an instrument for programmatic-level assessment of information literacy skills that is valid – and thus credible – to university administrators and other academic personnel.9 Since its development, SAILS has been widely administered with more than 300 institutions using the assessment during the past decade. Reports of the instrument’s use and studies of its efficacy and validity have been published in library literature.10,11,12,13,14 
Project SAILS administers the SAILS cohort test twice a year, during fall and spring. The SAILS cohort test is intended to measure the information literacy knowledge of groups of students and not serve as an assessment of the individual test takers. The test results for each institution can be benchmarked externally by comparing scores to those at other participating institutions and institutions of similar scope. The results can also be analyzed internally, as scores can be compared between students of different classifications, majors, and other locally determined variables. 
The SAILS test bank contains a total of 161 questions. Each test participant receives a randomly generated list of forty-five questions with specific questions covering each skill set. Most students complete the test within thirty-five minutes. Responses are stored in the central SAILS database where data are analyzed and reports are generated. SAILS questions cover the following eight research skills: 
• Developing a research strategy
• Selecting finding tools 
• Searching
• Using finding tool features
• Retrieving sources
• Evaluating sources
• Documenting sources
• Understanding economic, legal, and social issues of information.15
Project SAILS at UALR
In fall 2011, the university provided funding via a campus assessment grant to conduct Project SAILS. The SAILS instrument was chosen, as it was a widely-used, well-researched, and relatively inexpensive cohort test that provided the opportunity to conduct comparisons of UALR to peer institutions. The UALR Institutional Review Board (IRB) reviewed plans for the testing and provided clearance with the stipulation that the test be administered so that participation was voluntary and that the library would give no incentives to students for taking the test.
In conducting the SAILS assessment, the university focused on two populations – freshmen and seniors. In that way, the university would be able to determine the relative information literacy and research skills of samples of students at the beginning and end of their academic careers. The university could then determine the needs of incoming students as well as a measure of the skill attainment of UALR graduates.  
Academic deans from each UALR college identified senior-level classes to participate. Upon the suggestion of the associate vice chancellor for academic affairs and student success, and with the consent from the instructors, eight sections of the mandatory freshman first-year experience course were identified to participate in the assessment. While other students participated, most test-takers were from the first-year experience or senior capstone classes. 
The tests were administered online via Blackboard, the university’s course management system, as well as in the classroom setting. The students in the first-year experience sections completed the test during class time in the library’s computer classroom. The other students took the test online via Blackboard. Completion rates for senior-level classes via Blackboard were low due to the nature of online testing, as well as the general lack of incentives. Despite detailed instructions, there was some confusion among faculty regarding how students were to access the assessment through Blackboard. As a result, senior representation was less than expected and the test sample was strongly weighted toward freshmen, who accounted for 68 percent of UALR test participants.  
The administration of the test to freshmen is a common practice at other institutions participating in the SAILS assessment. As can be seen in Figure 1, freshmen at all institutions accounted for 55 percent of test takers in spring 2012; at doctoral level institutions, freshmen accounted for 78 percent. At 20 percent, the number of seniors taking the test at UALR was significantly smaller than the number of freshmen. However, this percentage was higher than senior test takers at all institutions (15 percent) and doctorate-level institutions (9 percent).

Figure 1: SAILS Participation by Student Classification

While the classification of students taking the test at UALR was disproportionately freshmen, the sample of test participants included representation that closely matched the university’s demographics. As can be seen in Figure 2, minority representation was slightly higher in the test population – 44 percent were male, 28 percent African American, and 5 percent Hispanic. 

Figure 2: UALR Test Participants by Gender, Race, and Ethnicity

In the administration of the assessment, Project SAILS electronically collects and tabulates each student’s responses and calculates scores using a one-parameter Rasch model measurement based on a combination of the degree of question difficulty and student performance. Test scores are placed on a scale from 0 to 1000, using test scores from all participating institutions to create a benchmark file for comparison purposes. Student success on each question is used to determine the difficulty level of each item, and student responses are analyzed to determine an average score for each participating cohort. 
The benchmark data is based on the score of all participating institutions, with each participating institution’s scores being compared to the scores of all institutions and the institutions of the same highest degree-level. During spring 2012, students from ninety-nine institutions participated. Of these institutions, eleven were associate degree-level, thirty-seven were bachelor-level, thirty-three were master-level, and eighteen were doctorate-level universities.
As can be seen in Table 1, SAILS results include an average score for the institution, the institution-level group (i.e., doctorate-level for UALR), and the all institutions group. A standard error is reported for each score. The standard error (indicated with ±) reflects the score variability that occurs due to a combination of sampling and measurement error. A smaller sample size results in a greater standard error and a lesser degree of accuracy. The actual average score falls between two numbers, calculated by adding and subtracting the standard error. For example, in Table 1, the UALR score of 469 ± 9 for skill set one, “Developing a Research Strategy,” indicates that the true group average score falls in the range of 460 to 478. Ranges of scores that overlap are not significantly different; those that do not overlap are significantly different. In skill set one, the UALR score is lower and the difference is statistically significant.
Overall, UALR students taking the test scored lower than students at other institutions in seven out of eight skill sets. UALR students were about the same as their peers in the area of “Understanding Economic, Legal, and Social Issues of Information.” Similarly, freshmen at UALR scored lower than freshmen at other institutions in seven out of eight skill sets and were about the same as other freshmen in the area of “Understanding Economic, Legal, and Social Issues of Information.”

Table 1: Freshmen Scores

Information Literacy Skill Set 
	
UALR Freshmen
	
Doctoral-Level Freshmen	
Freshmen at All Institutions
	
Developing a Research Strategy 
	469 (±9)	495 (±1)	490 (±1)	
Selecting Finding Tools 
	462 (±10)	502 (±1)	496 (±1)	
Searching 
	457 (±8)	480 (±1)	475 (±1)	
Using Finding Tool Features 
	487 (±12)	524 (±1)	521 (±1)	
Retrieving Sources 
	468 (±11)	506 (±1)	502 (±1)	
Evaluating Sources 
	446 (±9)	471 (±1)	467 (±1)	
Documenting Sources
	408 (±13)	472 (±1)	458 (±1)	
Understanding Economic, Legal, and Social Issues 	450 (±9)	462 (±1)	454 (±1)	

While UALR freshmen did not perform well in comparison to freshmen at other institutions, seniors did better. As can be seen in Table 2, seniors at UALR scored about the same as their peers at other institutions in all eight SAILS skill sets. Any difference in scores is not statistically significant.  

Table 2: Senior Scores

Information Literacy Skill Set
	
UALR Seniors	
Doctoral-Level
Seniors 	
Seniors at All Institutions
	
Developing a Research Strategy 
	534 (±16)	539 (±3)	523 (±1)	
Selecting Finding Tools 
	537 (±20)	542 (±3)	526 (±1)	
Searching 
	507 (±17)	520 (±1)	507 (±0)	
Using Finding Tool Features 
	561 (±21)	575 (±4)	557 (±2)	
Retrieving Sources 
	541 (±24)	578 (±4)	560 (±2)	
Evaluating Sources 
	492 (±20)	515 (±3)	500 (±1)	
Documenting Sources 
	523 (±23)	549 (±4)	516 (±2)	
Understanding Economic, Legal, and Social Issues 	483 (±18)	512 (±3)	490 (±1)	

While freshmen are underperforming, seniors are demonstrating skills equal to seniors at doctoral-level universities and other institutions. UALR students may be demonstrating a growth in skill level as they progress to graduation. As can be seen in Table 3, seniors outperformed freshmen in all measured skill sets.

Table 3: UALR Freshmen and Senior Comparison

Course 
	UALR Freshmen 	UALR Seniors 	
Class Difference 
	
Developing a Research Strategy 
	469 (±9)
	534 (±16)
	14%	
Selecting Finding Tools 
	462 (±10)
	537 (±20)
	16%	
Searching 
	457 (±8)
	507 (±17)
	11%	
Using Finding Tool Features 
	487 (±12)
	561 (±21)
	15%	
Retrieving Sources 
	468 (±11)
	541 (±24)
	16%	
Evaluating Sources 
	446 (±9)
	492 (±20)
	10%	
Documenting Sources 
	408 (±13)
	523 (±23)
	28%	
Understanding Economic, Legal, and Social Issues 	450 (±9)
	483 (±18)
	7%	

This difference in scores with freshmen being below and seniors being about the same as their peers at other universities nationally may be interpreted in two ways. On one hand, this indicates that the university may be providing educational experiences that develop students’ research skills and make students information literate with the capacity for lifelong learning. On the other hand, the test results may indicate that students with stronger research skills are better prepared to perform the work required for their degrees and to achieve student success. Both of these conditions are likely true. A longitudinal study of a specific cohort of students would be needed to make a clearer determination of the impact of the UALR undergraduate curriculum on student attainment of information literacy skills. 
In addition to student classification, the SAILS assessment reported scores by race/ethnicity, major, and transfer credits. Student major is a standard breakdown for all SAILS participants; race/ethnicity and the number of transfer credits were specific variables that UALR selected for study. While the assessment indicated that differences exist among the selected populations, student classification proved to be the most significant factor in determining a student’s skill level.  
Conclusion
The results of the assessment of students’ information literacy skills at the University of Arkansas at Little Rock indicates that students are arriving on campus without the research skills needed for academic success. UALR freshmen are behind their peers at other universities nationally in taking a research problem and developing a research strategy, identifying appropriate information resources, searching these resources, and then retrieving, critically evaluating, and documenting the published sources needed to address their research problem. As a result, the university needs to create educational experiences that lead to the development of these skills.
To support this endeavor, librarians at the UALR Ottenheimer Library are partnering with university initiatives to support at-risk students, such as the African American Male Initiative, African American Female Initiative, and the Hispanic Initiative. Librarians are also collaborating with teaching faculty to create research-intensive assignments and foster student skills by providing library instruction beginning at the freshman level and continuing throughout the curriculum.  The university’s faculty senate is currently revising the core curriculum and information literacy is one of a number of general education outcomes that is being considered for inclusion. The Project SAILS assessment has provided data that supports the need for information literacy skills in the university’s undergraduates.  By promoting the development of these skills the university will be able to better facilitate student success.

____________________________________
1 Association of College and Research Libraries. Information Literacy Competency Standards for Higher Education (Chicago: American Library Association, 2000). http://www.ala.org/acrl/sites/ala.org.acrl/files/content/standards/standards.pdf.

2 Ibid.

3   Laura Saunders, “Regional Accreditation Organizations' Treatment of Information Literacy: Definitions, Collaboration, and Assessment,” The Journal of Academic Librarianship 33, no. 3 (2007): 317-26.

4 Project SAILS, “About the SAILS Test,” accessed February 16, 2013.  https://www.projectsails.org/. 

5 ETS iSkills, “About the iSkills Assessment,” accessed February 16, 2013.  http://www.ets.org/iskills/about/.

6 Madison Assessment, “Information Literacy Test,” accessed February 16, 2013.  http://www.madisonassessment.com/assessment-testing/information-literacy-test/. 

7 Carol Leibiger, and William E. Schweinle, “The South Dakota Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy,” In Sailing Into the Future-Charting Our Destiny: Proceedings of the Thirteenth National Conference of the Association of the College and Research Libraries, edited by Hugh A. Thompson, 264-76 (Chicago: Association of College and Research Libraries, 2007).
8 Julia C. Blixrud, “Project SAILS: Standardized Assessment of Information Literacy Skills,” ARL:  A Bimonthly Report on Research Library Issues & Actions 230/231 (2003): 18-19. 
9 Lisa G. O’Connor, Carolyn J. Radcliff, and Julie A. Gedeon, “Applying Systems Design and Item Response Theory to the Problem of Measuring Information Literacy Skills,” College and Research Libraries 63, no. 6 (2002): 528-43.
10 Joseph A. Salem Jr. and Carolyn J. Radcliff, “Using the SAILS Test to Assess Information Literacy,” In Building Effective, Sustainable, Practical Assessment: Proceedings of the Second Library Assessment Conference, edited by Francine DeFranco, Steve Hiller, Lisa Janicke Hinchliffe, Kristina Justh, Martha Kryllidou, Jim Self, and Joan Stein, 131-37 (Charlottesville, VA: Association of Research Libraries, 2006).
11 Juliet Rumble and Nancy Noe, “Project SAILS: Launching Information Literacy Assessment Across University Waters,” Technical Services Quarterly, 26, no. 4 (2009): 287-98.
12 Brian Lym, Hal Grossman, Lauren Yannotta, and Makram Talih, “Assessing the Assessment: How Institutions Administrated, Interpreted, and Used SAILS,” Reference Services Review 38, no. 1 (2010): 168-86.
13 Jason Martin, “Investigation Of Factors Affecting Information Literacy Student Learning Outcomes Fails To Undercover Significant Findings,” Evidence Based Library & Information Practice 6, no. 2 (2011): 59-60.
14 Yvonne Mery, Jill Newby, and Ke Peng, “Assessing the Reliability and Validity of Locally Developed Information Literacy Test Items,” Reference Services Review 39, no. 1 (2011): 98-122.
15 Project SAILS, accessed February 16, 2013.