Some Indiana colleges revolt against survey: U.S. News’ peer assessments called too subjective

The influential U.S. News & World Report college rankings come out next month, a rite of summer that causes many college administrators to groan.

Some administrators in Indiana and elsewhere, in fact, have grown so disenchanted with the survey-and see it as so flawed-that they have decided they’ll no longer participate in at least part of it after this year.

Nearly 100 private schools nationwide-including DePauw University and Earlham College in Indiana-are pulling out of the peer-assessment portion of the survey, saying it’s too subjective to be valid. The assessment accounts for 25 percent of a school’s score. Earlham, which last year ranked 65th among U.S. liberal arts colleges, is going a step further, opting out of the entire survey next year.

The annual survey, which U.S. News began in 1983, covers a wide range of areas, including student retention, faculty resources and alumni giving. But it’s the peer assessment that especially rankles many school officials who are asked to rate about 260 institutions on a scale of one to five or “don’t know.”

“Any college president will admit they have insider knowledge of only a handful of schools,” said Ken Owen, director of media relations at DePauw, which last year ranked 48th among liberal arts colleges.

The peer survey goes to several officials at each school. In DePauw’s case, President Robert Bottoms and the school’s vice president for academic affairs chose to not fill it out this year.

“[Bottoms] had just gotten to a point where he thought it was silly,” Owen said. “He was just checking boxes.”

Many of the schools declining to participate in future peer-review surveys made the call after they’d already submitted this year’s survey. Those that pull out still will be evaluated by others, and receive a score in the category. Nor will opting out of the entire survey keep Earlham out of the rankings. Even so, the dissident schools hope that by taking a stand, they’ll spur change in the way the magazine puts together the rankings.

Stefanie Niles, DePauw’s vice president of admission and financial aid, submitted her form before the school changed its policy.

Niles, who’s been at DePauw 10 years, said she considered a school’s reputation and selectivity of its admission process as she checked boxes. Niles estimated she ranked about half the schools on the list and checked the “don’t know” box on the rest.

School officials “are being asked to render these judgments without a true model or scale by which to measure each institution,” she said.

The number of schools Niles ranked is typical, said Robert Morse, director of data research at U.S. News.

“Each rater only rates on average 56 percent of the schools,” he said. “We’re confident that the aggregate rating of all people put together places a valid impression of that school.”

But Indiana schools say that, beyond being subjective, the survey fails to capture what’s most important-issues like the quality of students’ experience and how well they are prepared for life after college.

Such issues might be better represented in a database being compiled by the Washington, D.C.-based National Association of Independent Colleges and Universities. The group-which takes no stand on the U.S. News rankings-in September will launch a free, Web-based service that will include information on student life, internship opportunities and other areas.

Six Indiana schools so far have opted to participate, including Butler University, Earlham and the University of Notre Dame-which ranked 20th in U.S. News’ ranking last year of national schools, the only Indiana institution to land in the top 50.

Even schools continuing to participate in the survey call the peer rankings suspect.

“At best, it’s a gross approximation,” said Rab Mukerjea, director of strategic plan assessment at Purdue University. Purdue ranked 64th last year among national universities.

“We’ve been talking about this same issue for years,” Mukerjea said. “U.S. News has not seen it as important enough to address despite knowing schools don’t think it’s credible.”

Morse isn’t convinced the peer assessment is the real issue with the survey. He thinks schools may be publicly arguing this issue because it’s easier than disputing how they ranked.

“If we did the ranking without the peer survey, [schools] could find something else about the ranking to complain about,” he said.

Many schools say that, despite the survey’s shortcomings, it serves a purpose.

“There are some flaws, but it is helpful to consumers,” said Roger Thompson, vice provost for enrollment management at Indiana University, which also will continue participating. IU ranked 70th among national schools last year. “U.S. News represents one way to measure schools.”

Ball State, Butler and Indiana State universities, as well as IUPUI, also will continue participating.