The "AYP" Blues

Each year, states release lists of schools and school districts that have not made "adequate yearly progress" (AYP) in raising student achievement. Schools and districts appear on these lists because they have fallen short of one or more of the annual test score targets and/or other performance benchmarks set by states to comply with the federal No Child Left Behind Act (NCLB).

This process of monitoring and reporting on adequate yearly progress is a central concept of NCLB. It is meant to highlight schools and districts that aren't performing as well as they should and to stimulate actions that will improve teaching and learning. To make AYP, schools and districts must:

• Ensure that a state-determined percentage of students in every major subgroup—including major racial and ethnic minority groups as well as low-income students, disabled students, and English language learners—scores at the "proficient" level on state achievement tests in reading and mathematics;

• Test at least 95 percent of the students in each subgroup and overall;

• Meet at least one other state-determined academic indicator—graduation rates for high schools and attendance rates or another indicator for elementary and middle schools.

The percentage of students expected to score at the proficient level must go up over time until it reaches a goal of 100 percent of students achieving proficiency by 2014. The grade levels tested for AYP also increase over time.

Schools that do not make AYP for one year are placed on a "watch list" and must improve their performance the next school year to avoid future consequences. Schools that receive funds under the federal Title I program face increasingly serious sanctions if they fail to make AYP for multiple years. After two consecutive years of falling short, these schools are identified for improvement. They must develop a specific plan to boost student achievement, and their school district must reserve a portion of its Title I money to finance transportation for students who choose to transfer to a higher-performing public school. After three or more consecutive years of missing the mark, their district must use a portion of its Title I funds to pay for tutoring services, and Title I schools are subject to stiffer sanctions, ending with replacement of administrative and/or teaching staff, restructuring of school governance, or takeover by a private management company or the state.

For three years, the Center on Education Policy (CEP) has been conducting a comprehensive national study of federal, state, and local implementation of the No Child Left Behind Act. This work has shed light on various reasons why a school or school district may fail to make AYP.* Simply put, low-achieving schools will fail to make AYP, but not all schools that fail to make AYP are performing poorly.

Why Could a School Fail to Make AYP?

Here are some of the most important reasons why schools miss AYP targets.^†

1. Some schools are doing a poor job of educating students.

The most obvious reason why some schools do not make AYP is because they are not adequately educating students. Often these schools fall short in several areas. They may have low test scores overall, as well as for specific subgroups of students, and they may also have low graduation rates or poor attendance rates. These are the kinds of schools that NCLB accountability was intended to highlight. For these schools, being identified for improvement can bring technical assistance, additional resources, and other supports to help them do better. It can also mean a loss of enrollments, if parents take advantage of the NCLB choice option, and a diversion of school district Title I funds to pay for student transportation to choice schools or off-site private tutoring services.

2. Schools with diverse enrollments often have a more difficult time making AYP than small or relatively homogenous schools because they must meet performance targets for more subgroups.

Even if a school's overall test scores exceed state AYP targets, the school will fail to make AYP if just one student subgroup misses the mark. Consequently, more diverse schools and districts have a tougher time making AYP than less diverse ones because they have more subgroups large enough to count for AYP (for more information on minimum subgroup sizes, see item three). CEP's study (2005) found that significantly more urban and very large districts had schools that did not make AYP for one or more years than did suburban, rural, or small districts. This is partly because urban and very large districts tend to have more subgroups that count for AYP.

Schools with subgroups may not make AYP because they aren't doing a good job educating students in certain subgroups. They may also be making some progress with their subgroups, but not enough progress to meet the AYP target (for more on this, see item four). Or, they may not make AYP simply because each additional subgroup creates an additional chance for test score fluctuations (see "Accountability 101: Tests Are Blunt Instruments").

3. Whether a school makes AYP depends on which state it is located in.

The proportion of schools not making AYP varies enormously from state to state. Based on 2003–04 testing, less than five percent of the public schools in Wisconsin did not make AYP, compared with about 76 percent in Florida (Wisconsin Department of Public Instruction, 2004; Florida Department of Education, 2004). At least some of this disparity is the result of different state policies for implementing AYP, rather than differences in student learning. Because of these differences, comparisons of state AYP lists are not a meaningful indicator of the quality of schools in one state versus another.

The No Child Left Behind Act lets states make several key decisions about their accountability systems. For example, each state:

• Determines its own curriculum standards;

• Develops or chooses its own tests to measure progress toward its standards;

• Decides where to set the cut scores on state tests to define "proficient" performance for AYP;

• Sets targets beyond the starting point^‡ for the percentage of students that must score at the proficient level each year, as long as these targets rise periodically and are set at 100 percent proficient by 2014;

• Decides how large subgroups must be in order to be counted for AYP;

• Decides whether to use statistical methods like confidence intervals to compensate for test score fluctuations stemming from factors unrelated to learning (see "Accountability 101");

• Selects and sets standards for determining high school graduation rates and, in grades three through eight, chooses another indicator of performance, such as attendance.

Together, these decisions affect how easy or hard it is for schools to make AYP.

One area of state difference relates to test difficulty, test cut scores, and targets for the percentage of students reaching proficiency. Some states use less rigorous tests while others administer harder tests. Some states have lower cut scores for proficient performance while others have higher ones. In 2004–05, Tennessee fourth-graders had to correctly answer 36 percent of the items on the state test to achieve the proficient level in language arts, while New York fourth-graders had to get 67 percent right (Tennessee Department of Education; New York State Department of Education, 2005). Some states have set steadily rising targets for the percentage of students that must score at proficient levels on the way to 100 percent proficiency, while other states have "backloaded" their trajectories, setting relatively low targets in the early years and calling for faster progress in later years. By the law's formula, Oregon, for example, had a starting target of 39 percent proficient in math in 2002–03 and chose to increase it to 49 percent in 2004–05 (Oregon Department of Education). Virginia had a starting target of 58.4 percent proficient in math in 2002–03 and chose to increase it to 70 percent proficient in 2004–05 (Virginia Board of Education). Although some states with higher cut scores have lower proficiency targets and vice versa, this is not always the case; the interactions of test difficulty, cut scores, and proficiency targets do not seem to follow a clear pattern. Studies have shown, however, that students with exactly the same knowledge and skills would miss the proficiency target in some states and easily surpass it in others (Helderman and Mui, 2003).

State differences in minimum subgroup size also influence whether a school makes AYP. In New Mexico, a subgroup must include at least 25 students to be separately counted for AYP purposes, while in Virginia the minimum size is 50 students (New Mexico State Department of Education; Virginia Board of Education). During the past year, several states increased their minimum subgroup size, in effect making it easier for schools to meet AYP targets because fewer schools have subgroups large enough to count (CEP, 2005).

4. A school could raise achievement for struggling students but still not make AYP if these students score below the proficient level.

Rather than tracking the progress of individual students, AYP is generally based on the percentage of students reaching a fixed proficiency target. Students who score far below the proficient level have much more ground to cover than students who are closer to proficiency. But with few exceptions, schools do not receive AYP credit for improving the achievement of these lowest-performing students. Nor do they get credit for raising the achievement of high-performing students who have already reached proficiency.

Consider two hypothetical schools (portrayed in the figure on the right) in a state with an AYP target of 50 percent proficient in math for 2004–05, rising to 70 percent proficient in 2007–08. A school in which 70 percent of students already score proficient could post no gains (or could even show declines) for a few years and still make AYP through 2007–08. But a school in a poor neighborhood that steadily boosted its achievement from 25 percent proficient in 2004–2005 to 40 percent in 2007–08 would fail to make AYP. In short, the first school could stagnate, while students and teachers in the second school could easily become discouraged because their gains don't count. The same situation could arise with high-performing and low-performing subgroups.

NCLB's "safe harbor" provision offers an exception to this principle by giving credit for significant improvements below the proficient level. If a school does not meet state AYP targets, but it reduces by 10 percent the share of students scoring below proficient and also makes progress on the state's other academic indicator, then it will make AYP under safe harbor. For example, if 75 percent of the students in a high school score below proficient one year, and 67.5 percent score below proficient the next year, and the graduation rate increases sufficiently, the school will make AYP. Even so, safe harbor is a challenge because each major subgroup that misses the fixed AYP targets must separately meet the 10 percent improvement criteria to qualify for safe harbor.

Exceptions also exist in the limited number of states that use a performance index as part of their NCLB accountability systems. These states, which include Massachusetts, Minnesota, New York, and a few others, give schools or districts partial credit toward AYP for partially proficient students (American Federation of Teachers, n.d.). Minnesota, for example, assigns half an index point for each student reaching the partially proficient level and one full point for each student reaching the proficient level. These indices may help some improving schools make AYP that otherwise would have failed. Only a handful of states have sought approval for index systems from the U.S. Department of Education.

[[{"type":"media","view_mode":"wysiwyg","fid":"934","attributes":{"alt":"High-Achieving vs. High-Performing Schools","title":"","class":"media-image","typeof":"foaf:Image","wysiwyg":1,"width":"350","height":"438"}}]]

5. AYP hurdles become more numerous and higher over time.

By 2006, students must be tested yearly in reading and math in grades 3 through 8 and once during grades 10 to 12. By 2007–08, students must be tested in science, too, although states can decide whether to include science results in their AYP calculations. Many states are still phasing in all the NCLB-required tests. In 2004–05, about half the states were administering reading and math tests in all the required grades, and 23 states were administering science tests (Olson, 2004). The expansion of testing to additional grades in many states and to another subject in some states could create more opportunities for schools to miss AYP targets.

Even more significantly, the AYP hurdles get higher over time as states move closer to 100 percent proficiency. Given the normal fluctuations that occur in yearly test scores (see "Accountability 101: Tests Are Blunt Instruments"), it is unrealistic to expect every school and district to post steady achievement gains year after year. In addition, several states and districts participating in CEP's study (2005) thought that the requirement of 100 percent proficiency was unrealistic, particularly for the subgroups of students with disabilities and English language learners. Whether or not schools surmount the current set of AYP hurdles, the bar continues to rise. An analysis of AYP trends in Connecticut (which is consistently one of the highest-scoring states on the National Assessment of Educational Progress) concluded that as expectations are raised and more grades are tested, 90 percent of the elementary and middle schools in the state will fail to meet AYP targets by 2014 (Frahm, 2004). Other states have projected similar outcomes.

6. AYP is determined largely by looking at whether the students in the tested grades of a school reach fixed proficiency targets each year. Changes in the composition of the test-taking population could affect the ability of one year's group to meet the fixed targets, especially for smaller schools or subgroups.

The composition of a class, school, or subgroup often varies from year to year in ways that can affect test scores. Students move in and out of the district. One year's group could have an unusually high number of English language learners due to an influx of immigrant children. The loss of a major manufacturer could mean higher poverty and more family disruption. These year-to-year changes can make AYP determinations unreliable, especially for smaller schools or subgroups or in states that do not apply statistical tools such as confidence intervals (see "Accountability 101").

What Does This Mean for Schools and Districts?

All of the factors mentioned above, plus others not cited here, create a long, difficult course with many hurdles that schools and districts must traverse to make AYP. Moreover, the sanctions are the same for schools that miss by a little or miss by a lot. For this reason, some educators, researchers, and policymakers have concluded that the current AYP requirements are unrealistic or unfair.

Nonetheless, the NCLB requirements have produced some positive outcomes. They have pushed schools and communities to pay more attention to children and subgroups that aren't achieving as well as they should. The requirements also seem to be spurring schools to take concrete actions to improve student achievement (CEP, 2005).

The complexities of AYP have important implications for schools and districts. First, many parents and other citizens do not realize the various reasons why schools could miss AYP targets and may unfairly assume that something is seriously wrong with their local school, even when that's not the case. Clearly, the AYP process will identify a lot of low-achieving schools, but just because a school has been identified doesn't make it low performing. Second, if too many schools fail to make AYP, it can undermine the credibility of the federal requirements and create incentives for states to lower cut scores or dumb down their tests. Third, if too many schools are cited as needing improvement, states and districts will not have enough resources, staff, time, and expertise to help the schools with the most serious educational problems.

The intent of No Child Left Behind is to greatly increase learning for all groups of students by highlighting schools and subgroups with legitimate educational needs and marshalling resources to boost their achievement. Underperforming schools need technical and financial assistance to carry out proven strategies for improving student learning. Over the long term, the President and the Congress should consider revising the adequate yearly progress requirements of NCLB based on evidence from CEP's study, other research, and the experiences of states and school districts.

---

Nancy Kober is a consultant to the Center on Education Policy and co-author and editor of the Center's annual reports on NCLB. This article is an adaptation of the September 2004 issue of CEP's Test Talk for Leaders. The full version is available in CEP's Web site at www.cep-dc.org.

Endnotes

*As this article goes to press, the U.S. Department of Education has announced that states will be given greater flexibility, so some of the issues presented here may be addressed by changing regulations. (back to article)

^†For a more comprehensive discussion of reasons why schools fail to make AYP, see www.cep-dc.org/pubs/TestTalk/CEPTestTalk3.pdf. (back to article)

^‡Each state's starting point is calculated through a formula in the NCLB law. (back to article)

References

American Federation of Teachers (n.d.). Use of performance indexes. Unpublished paper.

Center on Education Policy (2005). From the capital to the classroom: Year 3 of the No Child Left Behind Act. Washington, D.C.: CEP.

Florida Department of Education (August 17, 2004). "School grades, adequate yearly progress, and school recognition." Commissioner's presentation to the State Board of Education. Retrieved from www.fldoe.org/meetings/2004_08_16/SchoolGrades_Pres.pdf.

Frahm, R. (May 28, 2004). "Study predicts school failures." Hartford Courant.

Helderman, R. S. and Mui, Y. Q. (September 25, 2003). View on schools' progress unclear. Washington Post.

New Mexico State Department of Education (n.d.). Consolidated state application accountability workbook. Retrieved from www.ed.gov/admins/lead/account/stateplans03/nmcsa.pdf.

New York State Department of Education (February 2005). "Grade 4 English Language Arts (ELA) Test." Retrieved from www.emsc.nysed.gov/osa/concht/05/gr4ela.htm.

Olson, L. (December 8, 2004). "Taking root." Education Week.

Oregon Department of Education (n.d.). "Key points about adequate yearly progress (AYP)." Retrieved from www.ode.state.or.us/initiatives/nclb/docs/aypkeypoints.rtf.

Tennessee Department of Education (n.d.). "Proficiency ranges for TCAP achievement 2004–2005 CRT scores." Retrieved from www.state.tn.us/education/assessment/tsach03nproflvl.php.

Virginia Board of Education (n.d.). Consolidated state application: Amended accountability workbook. Retrieved from www.ed.gov/admins/lead/account/stateplans03/vacsa.pdf.

Wisconsin Department of Public Instruction. (September 17, 2004). "Preliminary AYP notification sent to schools and districts." News release.

Related Articles

The "AYP" Blues
Low-Achieving Schools Will Fail—but They're Not the Only Ones
By Nancy Kober

Accountability 101: Tests Are Blunt Instruments

Ideas to Consider: How to Make AYP Work Better for Students and Their Schools