Originally Posted by
heffray
Here’s a transcript of the interview Malcom has with Robert Morse, the guy in charge of the US News rankings:
Gladwell: Describe again, how is the reputation score generated?
Morse: So we send out three surveys per school and then they respond. They rate the schools on a scale of five to one or no response.
Gladwell: So who are the three people at each school who are asked to fill out the survey?
Morse: The president, the provost, and the enrollment manager or the head of admissions.
Gladwell Narrating: The surveys arrive via email once a year to everyone who matters in American higher ed. If you're at a liberal arts college, you're given a list of all 222 other liberal arts colleges in your category. If you're part of a big university, you get a list of the 388 other big universities in the United States. Your job is to rank the schools on your list on a scale of one to five, five being amazing, one being there's a serious problem here.
Gladwell: And on what basis are people making those judgments?
Morse: Well, I mean, that's a good question. I mean, we asked them to base it on their view of the school's reputation for undergraduate academic quality.
Gladwell: But I mean, how would you know?
Morse: I mean, we believe that the aggregate sum of all the raters who are leaders in higher education, you know, they're presidents and provosts and mission deans, so they're not rookies. When you aggregate their views at any given time, they're representing where those schools stand in the marketplace.
Gladwell: I mean, so its an assessment of their feelings about other schools.
Morse: Well, they're leaders in higher education. So it's based on their knowledge of the other schools that they've gathered or through meetings or for exposure. Do they necessarily know a lot about all the schools they're rating? I don't know.
Gladwell: If I rate a restaurant on Yelp, it's because I've eaten there. Right?
Morse: Yes.
Gladwell: But here, when I rate a university on the US News Ranking, it's not because I've attended that university or taught there. It's just—
Morse: Well, you may have. We shouldn't say that you didn't. But if you're saying, have the people who rated the schools been on every campus or taught at that school or know in depth their course catalog? No. Do some of them have greater degrees of knowledge about the schools they're rating? Yes. Do some of them have lesser degrees of knowledge? Yes. So I think it varies from having substantial knowledge to far less knowledge.
…it goes on from there. But I think you get the gist.