Accountability Systems, Surveys and Assessment Tools

In response to calls for increased institutional accountability, transparency and ease of comparing institutional information and students' outcomes, many of the large educational associations and organizations have created databases, accountability systems, surveys of student engagement, and learning outcomes assessment tools. We profile some of these instruments on this page—and we'd like to hear about your experiences with these tools and any others we may have overlooked.

Federal Data

Many of the tools profiled below draw on federal data provided for public use. We address these tools first.

The Integrated Postsecondary Education Data System (IPEDS)

The federal database commonly referred to as IPEDS is the primary source for data on colleges, universities, and technical and vocational postsecondary institutions in the United States. The Higher Education Act of 1965, as amended, requires that institutions that participate in federal student aid programs report data on enrollments, program completions, graduation rates, faculty and staff, finances, institutional prices, and student financial aid. These data are made available to students and parents through the College Navigator college search Web site and to researchers and others through the IPEDS Data Center.

IPEDS is widely considered to be an inadequate data collection system. The federal graduation rate that relies on IPEDS data is only able to track first-time, full-time freshmen, who comprise only a portion of the higher education student body—part-time and transfer students are excluded.

Click here to visit the IPEDS Web site.

College Navigator

Formerly known as the College Opportunities Online Locator (COOL)—but revamped and renamed in 2007—College Navigator is administered by the National Center for Education Statistics (NCES). Users can search and compare data for four-year, two-year, public and private colleges and universities. Drawing from the federal data set Integrated Postsecondary Education Data System (IPEDS), users can compare, side-by-side, institutions' admissions rates, financial aid offered, SAT requirements, retention and graduation rates, majors, athletic teams, and information about campus security—as well as accreditation status.

Click here to visit College Navigator's Web site.

Accountability Systems

These are the accountability systems developed by national associations in response to the drumbeat of calls for increased transparency, accountability, and comparability for institutions of higher education.

These systems of accountability can be adopted by institutions as a means toward complying with the Voluntary System of Accountability.

College Choices for Adults

Transparency By Design's College Choices for Adults Web site is an accountability framework aimed primarily at adult online learners and designed to make institutions—usually those providing distance education—easily comparable. Each participating institution provides data on program-level learning outcomes and results of outcomes assessment as well as institutional data including results of the National Survey of Student Engagement (NSSE), the ETS Proficiency Profile (formerly MAPP), Priorities Survey for Online Learners by Noel-Levitz, and common alumni surveys for public display on the website. Potential adult learners can utilize this data to assess their options for learning at a distance based on how each program's learning outcomes match their personal and professional goals.

The participating institutions are a mix of for-profit, private not-for-profit, and public institutions, and they include: American InterContinental University Online, American Public University System, Capella University, Charter Oak State College, Excelsior College, Fort Hays State University, Franklin University, Kaplan University, Regis University, Rio Salado College, Southwestern College Professional Studies, Western Governors University, and Union Institute and University. Given that this group of institutions varies widely with regard to institutional mission, comparing individual programs or majors is difficult. To remedy this, each program is allowed to develop their own goals for student learning outcomes. The institution publishes each program's individualized student learning outcomes along with their students' progress towards these outcomes.

Click here to visit the College Choices for Adults Web site.

College Portrait

College Portrait was developed in 2007 by members of the Association of Public and Land Grant Universities (APLU) and the American Association of State Colleges and Universities (AASCU) as part of the Voluntary System of Accountability to increase comparability, demonstrate accountability, and measure educational outcomes for public four-year institutions. Nearly 300 institutions are now participating in the project that allows users to compare institutions' statistics on the student body, tuition costs, graduation and retention rates (a new formula described below), and learning outcomes.

For measuring graduation rates, College Portrait uses a "student success and progress rate," which attempts to provide more useful statistics that the federal graduation rate does not provide, such as tracking transfer students. College Portrait provides information on how many students graduate within four or six years, how many are still enrolled, how many have graduated from another institution, and how many are still enrolled at another institution.

The student learning outcomes section has come under fire by many participating institutions because it requires a two-part testing component. In one part, colleges will be required to provide data from state licensure examinations and other tests taken by graduates of certain programs, so that prospective students can see, for example, the passage rates on the tests required to become a nurse or a teacher in the state. In part two, schools can either publish their own student learning goals and progress made toward these goals using institution-specific data and program assessments, or can display results from national student learning outcomes assessment instruments such as CAAP, CLA, or MAPP surveys. Students would take the exams as freshmen and seniors so that results would indicate both test scores and gains during undergraduate education. The student learning outcomes section is currently a pilot program, but participation is strongly encouraged.

Institutions must also select one of four student experience surveys to administer: the College Student Experiences Questionnaire, the College Senior Survey, the National Survey of Student Engagement or the University of California Undergraduate Student Experience Survey then publish their findings on their individual college portrait.

Click here to visit the College Portrait Web site.

Measuring Up

Conducted biennially by the National Center for Public Policy and Higher Education, Measuring Up is a national report card that evaluates the progress of the nation and all 50 states in providing Americans with education and training beyond high school through the bachelor's degree.

The report card grades in six overall performance categories:

• Preparation: How adequately does the state prepare students for education and training beyond high school?
• Participation: Do state residents have sufficient opportunities to enroll in education and training beyond high school?
• Affordability: How affordable is higher education for students and their families?
• Completion: Do students make progress toward and complete their certificates or degrees in a timely manner?
• Benefits: What benefits does the state receive from having a highly educated population?
• Learning: What is known about student learning as a result of education and training beyond high school?

You can view the most recent report, Measuring Up 2008, by clicking here.

University and College Accountability Network (U-CAN)

U-CAN is a Web site developed in 2007 by the National Association of Independent Colleges and Universities (NAICU) to provide information to prospective college students and their parents. U-CAN, which profiles approximately 800 private institutions, was developed in response to the findings of the Spellings Commission. Institutional profiles do not include student learning outcomes measures—NAICU claims that in focus group testing, there was little demand for this information, and that the lack of a universal assessment standard for student learning makes comparison difficult. Instead, institutions can link to any assessments or student learning outcomes measures they use on their profile. Users can also learn about institutions' admissions, enrollment and graduation rates as well as tuition amounts and average undergraduate class sizes.

U-CAN promotes itself as a forward-thinking Web site using what they call "new media." By advertising through targeted Google and Facebook ads, a Wikipedia page, YouTube videos, and other social networking sites, U-CAN strives to reach the young and distance themselves from US News and World Report-style published ranking systems. The U-CAN Web site also has a blog component.

Click here to visit U-CAN's Web site.

Voluntary System of Accountability (VSA)

Proposed by the Association of Public and Land Grant Universities (APLU) and the American Association of State Colleges and Universities (AASCU) for the Spellings Commission, the VSA answers demands from the commission and state legislatures for better measures of institutional and student performance. The VSA encourages institutions to compile information including price, living arrangements, graduation rates and curricula, and also report results of measures of student engagement (such as NSSE) and measures of student learning (such as CLA, CAAP or MAPP). College Portrait, described below, is an accountability system that was developed for institutions to combine all of this information into one central location for public consumption and compliance with the VSA.

For more on the VSA, click here.

Measures of Student Learning Outcomes

Tests that measure student learning outcomes typically assess specific skills that students are expected to gain from their education, such as writing skills, reading comprehension, mathematical concepts, etc. Three of the most prominent measures of student learning are:

Collegiate Assessment of Academic Proficiency (CAAP)

One of the three major surveys of student learning, the Collegiate Assessment of Academic Proficiency offers six 40-minute test modules (reading, writing skills, writing essays, mathematics, science, and critical thinking) in a multiple-choice format (except for the writing essays module). Institutions can customize their assessment program by selecting modules that best reflect their mission and the goals and curricula of their general education programs. Built by the ACT, a CAAP score can be linked back to an ACT score received in high school to provide a value-added assessment. Additionally, ACT suggests that CAAP be administered to a group of underclassmen and a group of upperclassmen to provide a value-added assessment. Institutions using CAAP are encouraged to develop benchmarks for their students, and CAAP can provide national or regional averages.

Click here to visit the CAAP Web site.

The Collegiate Learning Assessment (CLA)

The Collegiate Learning Assessment (CLA) is one of the three most-recognized standardized tests that measure learning outcomes—specifically, reasoning and communication skills —to determine how the institution as a whole contributes to student development. CLA gathers data that enables institutions to be compared with one another; small, representative groups of freshmen and seniors take the test to measure growth in learning.

Unlike CAAP and MAPP, the CLA utilizes a "value-added" model, i.e., the value-added of a student attending the institution, through assessing performance tasks and analytic writing tasks covering critical thinking, analytic reasoning, written communication, and problem solving.

The CLA is currently being adapted by the OECD Assessment of Higher Education Learning Outcomes (AHELO) in a study to assess the feasibility of capturing learning outcomes on an international scale by creating measures that would be valid for all cultures and languages. For more information, visit OECD's web page.

The CLA was built by RAND Corp's Council for Aid to Education (CAE) and has significant support from large foundations including Carnegie, Ford, Hewlett, and Lumina. In addition, the Community College Learning Assessment (CCLA) is an adaption of the Collegiate Learning Assessment for community colleges developed by the Council for Aid to Education and RAND Corporation.

Click here to visit the CLA Web site.

Measure of Academic Proficiency and Progress (MAPP)

One of the three major surveys that assess student learning, ETS' MAPP assesses students in three areas—writing, critical thinking/reading, and mathematics—with a single multiple-choice test lasting either 40 minutes (abbreviated version) or two hours. Participating institutions can add up to 50 locally authored multiple choice questions, an essay question, and nine demographic questions. Results place students in one of three levels of proficiency for each subject and also calculates norm-referenced scores.

Click here to visit the MAPP Web site.

***Note: ETS has renamed MAPP as the ETS Proficiency Profile.

Alternative measures of student learning outcomes:

LEAP/VALUE

The Association of American Colleges and Universities (AAC&U) has developed the Liberal Education and America's Promise (LEAP) campaign and the Valid Assessment of Learning in Undergraduate Education (VALUE) program to encourage colleges to use a broad array of learning measures, including electronic portfolios.

The LEAP campaign is organized around a set of "Essential Learning Outcomes" (pdf), which are described in College Learning for the New Global Century (pdf). These essential learning outcomes and a set of "Principles of Excellence" (pdf) provide a new framework to guide students' cumulative progress through college.

The VALUE project seeks to contribute to the national dialogue on assessment of college student learning. It builds on a philosophy of learning assessment that privileges multiple expert judgments of the quality of student work over reliance on standardized tests administered to samples of students outside of their required courses. The assessment approaches that VALUE advances are based on the shared understanding of faculty and academic professionals on campuses from across the country.

Measures of Student Engagement

Indiana University's College Student Experiences Questionnaire (CSEQ)

Indiana University developed the College Student Experiences Questionnaire (CSEQ) to gauge the amount of effort students used in accessing institutional resources and educational opportunities while attending college, based on strong research showing that there is a strong relationship between college learning and the quality of student effort and engagement. The survey, developed by George Kuh (founder of the National Survey of Student Engagement, or NSSE), has been used by more than 130 colleges and universities.

Institutions can use student response data to determine program effectiveness and the success of year-specific programs (senior portfolio, first-year classes), compile data for accreditation purposes, and gauge the effectiveness of student and academic affairs offices. The data provided by the questionnaire provides student and alumni service departments with a summary of the academic and extracurricular activities in which students engage while at a given institution.

The CSEQ is longer than some other questionnaires, raising concerns about overall response rates. When mailing questionnaires to selected freshmen and senior students, the University of California-San Diego calculated only a 37 percent response rate in 2000. New Century College (NCC) and George Mason University (GMU) compared administration methods in a 2003-2004 experiment. NCC students were given the questionnaire during class resulting in response rates of 79 percent for freshmen and 78 percent for seniors. George Mason mailed select freshmen and seniors questionnaires to the tune of 11 percent and 13 percent response rates respectively, despite multiple follow-ups and a variety of prize offerings.

Click here to visit the CSEQ web site.

National Survey of Student Engagement (NSSE)

This survey for students attending four-year colleges measures five areas of performance: level of academic challenge, active and collaborative learning, student-faculty interaction, enriching educational experiences, and supportive campus environment. It attempts to establish how students spend their time and what they gain from higher education. More than 1,300 colleges in the U.S. and Canada participated in NSSE.

Although only aggregate data is shown to the public (of course a participating institution can publish as much or as little of their results as they see fit), NSSE calculates more than just an institution-wide average score. Each college receives customized results compared with the aggregate data from their geographic region, other public/private institutions, and all NSSE participants from that year.

There are several similar tests to NSSE that are aimed at slightly different groups: the Faculty Survey of Student Engagement (FSSE), the Community College Survey of Student Engagement (CCSSE), the Law School Survey of Student Engagement (LSSSE), and the Beginning College Survey of Student Engagement (BCSSE) which is taken by incoming college freshman prior to the start of fall classes. Although each test is tailored to the group in question, the five areas of education performance and format are the same.

Click here to visit the NSSE Web site.