What Does It Mean When Schools Leave USNWR?

Session 520

Dr. Bryan Carmody joins us to discuss what effect schools leaving the USNWR ranking system will have on medical school applications and medical school admissions at large.

For more podcast resources to help you with your medical school journey and beyond, check out Meded Media.

Listen to this podcast episode with the player above, or keep reading for the highlights and takeaway points.

Dr. Bryan Carmody is a pediatric nephrologist, who was also on the show back in Episode 475. He talked about the unique obstacle that osteopathic medical schools may face in the future. He had a great YouTube video all about it. You can find him on Twitter if you follow @jmcarmody.

In this episode, Bryan will talk about US News and World Reports and what it means when medical schools start pulling out of participating in those rankings. Are the rankings going away? What’s going to happen?

[01:33] The MCAT Minute

The MCAT minute is brought to you by Blueprint.

Most students who take the MCAT have a free account. They use Blueprint resources every year, year in and year out.

If you have not created a free account over at Blueprint MCAT yet, go ahead and create one. You can get access to a half-length diagnostic, a free full-length, and their spaced repetition flashcard platform. Most importantly, you can use their free study planner tool because you can’t dwell on the MCAT if you don’t have a good plan.

[02:33] Withdrawal of Med Schools from US News and World Report Rankings

Bryan shares that he was not shocked by this news regarding the removal of participation by some medical schools from rankings. He says there has been a simmering resentment to the US News and World Report rankings among some of the medical schools for a while.

Some of the medical schools have declined to participate in the rankings for various reasons. It was just the ones who were winning in the rankings who decided to drop out this time.

[03:34] How Rankings Matter

The news on med school rankings removal has caused some rumblings and Bryan thinks the reason why is that rankings do matter. He says we have to acknowledge that it matters to applicants and the way that people perceive the school.

“It's a high-stakes competition, especially if your school is really particularly interested in the way that the average guy on the street perceives the school.” Click To Tweet

You can’t rise to the top of the US News and World Report rankings just by chance. You have to put in deliberate efforts to make sure that your school is excelling on the metrics that are in its formula. You have to continue to put in that effort, year after year in order to maintain that position. At the same time, other schools are doing that too. So it becomes a significant input for the school administration.

Bryan says, ultimately, the calculus was that the input that they are putting in did not improve their output. So if you are one of the schools that left and people are not thinking that you are the best school, there would be nothing for you to gain by participating in a competition that might convince them otherwise.

It is, however, not an issue of medical schools struggling to fill their seats. There are plenty of applicants, even more applicants than we have seats for medical schools here in this country. It is more on the applicant’s part where it matters. Someone who can beat their chest saying they got into the top 20 med schools.

[05:58] US News and World Report’s Methodology

A lot of people would assume that it must mean something spectacular if a school is out there as number two or number one on the US News and World Report. But when you break down the US News and World Report’s methodology, it is nothing more than a popularity contest.

Bryan says a big portion of it is the survey data from administrators and program directors.

“But in reality, very few of those people have any realistic foundation upon which to judge the schools.” Click To Tweet

They don’t know what’s going on at the schools. They don’t know the educational inputs or outputs except in the very broadest sense. In Bryan’s honest opinion, most of those people informed their voting on those things by the US News and World Report rankings. It becomes a self-fulfilling prophecy of circular logic.

[10:44] MCAT Scores as Metrics for Higher Ranking

One of the biggest concerns with US News and World Report methodology is how administrators are taking the time to figure out how to rank higher. It would seem to be working against the applicants and what is in their best interest. Ultimately, it had zero impact on education quality except for one variable, which is the faculty-to-student ratio.

By participating in it and by wanting to rank higher, schools want to have higher MCAT and GPA scores. It is not because they have data that shows higher MCAT scoring students do better for patient care or do better in their curriculum. It’s just so that they can have a higher rank on US News and World Report.

Bryan thinks the use of the MCAT is especially pernicious. Even if schools tried to abstain from it and look for a certain kind of student and an MCAT score that’s above a certain threshold, if your peer institutions continue to recruit students with higher MCAT scores, your ranking is going to go down.

Whatever your MCAT score median is, it is converted to a percentile scale mathematically, that is used in the rankings. In effect, even if your MCAT score stays the same if your peer institutions were accepting people with higher MCAT scores, your ranking can drop.

“The problem is not necessarily with rankings, it is with the interpretation of rankings and with the metric chasing that rankings inspire.”Click To Tweet

[16:33] Different Schools, Different Missions

Bryan says one of the fundamental flaws of US News and World Report rankings is the unspoken assumption that all schools are trying to do the exact same thing when in reality they’re not.

It may be true that what Harvard and Johns Hopkins are trying to do is substantially similar. But the first thing that you have to recognize is that different schools have different missions. And then the next thing that you have to look at is the admission variables that are there that will lead to the outputs that you want.

Bryan shares what they have been doing lately with their residency program where he works as an Associate Program Director. He says, one of the reasons that they exist is they want to train pediatricians who serve in the local area, those serving the kids around there, get specialty training, and come back.

He said they have done a pretty good job with a good number of their residents by doing that. They look at things such as how good they were at predicting that on the front end. Bryan thinks that it is a justifiable mission-driven admission process or recruitment process.

[18:16] School’s Mission and Baseline for Admissions

There is no doubt that we need some baseline level of competency to get through the admission process. There is comfort in using hard metrics for applicants because it gives a blueprint of what you should achieve and leaves out the uncertainty.

It does the students and the medical schools no good if they don’t accept students that are academically qualified. Students need to be able to demonstrate in one way or another, whether it’s GPA or MCAT, that they have the academic horsepower to succeed in that tough environment.

And beyond that, the schools should focus on the reason why the school exists. What are they here to do? Is the school’s primary mission just to be a social climber and climb these ranking systems? Or are they actually here to provide something different to society at large?

[22:49] Changes in MCAT Scoring

When asked if he would ever see the MCAT going to a pass/fail system, Bryan said he would not be surprised if the scoring changed at the higher end where there would be less numeric discrimination.

MCAT is intended to predict how likely you are to succeed in the first two years of medical school. If you look at the predictive validity of the MCAT, someone who has a 526 and a 520, probably doesn’t have any difference in their outcomes.

And yet, if we have systems like US News and World Report giving you a percentile, you will have a strong incentive to take someone with a 518, rather than a 514, or a 520 rather than a 517. Those numeric differences may not confer much more predictive validity because you are already at the flat end of the curve for the outcomes you care about.

In his opinion, Bryan says it is not going to be something that catches people by surprise if AAMC at some point will move to a way of reporting scores, such that high scores are reported as greater than something or other.

They are running a business. They would not do that unless they had a good belief that the Admissions Deans would be okay with that and if that’s what they wanted and would be useful to them.

[25:53] Data Association and Correlation with Outcomes

There is so much data that we are gathering as a society with technology these days. If you are someone with 10 years of practice as a physician, we can see your patient satisfaction scores, your outcomes, your readmission rates, and all the data related to you as a practicing physician, based on your specialty. And it can be tied all the way up to a score on your MCAT.

Asked if he can see a world where we can try to figure out some correlations of all these, Bryan says there are entities that are interested in creating that kind of data.

Generally, entities that offer some of the testing involved are very interested in tying USMLE scores to patient outcomes or similar things. So it can be done. When it’s done, you will have a very wide confidence interval on your estimate.

You might be able to show an association between someone’s USMLE score for example, with their patient mortality rate for some condition in the hospital years later. But it doesn’t mean that all the points on that estimate fall on a single line. There may be quite a bit of spread such that it negates the real interpretive value of the existence of a statistical difference.

Moreover, it will always include confounding because it’s not that these scores exist in a vacuum.

“Someone with higher SAT scores and lower SAT scores doesn't just differ in terms of their SAT score, they differ in whatever life experience led up to them taking the test.”Click To Tweet

More importantly, the SAT score itself sets in motion other things that may impact the variable that will have real impacts on the outcome that you’re interested in measuring.

[29:52] Staying in the Rankings

All of the schools have not removed their participation from the rankings.

“For each of the schools that have removed themselves, they have made the decision that staying in the rankings was more expensive or more constraining than it was valuable to them.”Click To Tweet

All the schools that have chosen to stay in have made the opposite decision. They have weighed the same data and have come down on the other side of it. They believe they will get better applicants and more applicants will ease their faculty recruiting. It would improve their name recognition for the hospital rankings that they really care about.

Whatever the side benefits of participating are, they have chosen that side of it.

[30:52] Expectations with the Rankings Reports

Bryan thinks it’s reasonable to expect to see more reports.

It’s not going away and they’re going to continue to recur and rank things. They will have to reconfigure the methodology somewhat in a way that will punish the schools that have tried to withdraw. For Bryan, this may sound cynical, but he does not believe that they will do that in a way that would call into question the authority of their rankings.

In his opinion, they will configure their formulas to benefit the schools that have maintained submitting data. So it is reasonable to assume that if your school is in the teens or the 20s, you might have a chance of rising a few spots as other schools have slid a little bit.

[31:55] Winners and Losers

The individual schools, whether they choose to stay or where they choose to go, view themselves as winners. They have made a decision that is right for them.

Bryan thinks students neither win nor lose. If you were a premed student, who had been socially isolated, completely unaware of what schools people think are the best medical schools, the US News Report rankings would provide a useful piece of knowledge. Otherwise, they just recreate the rankings that everybody already had in their head.

For the premed who wants to go to a top 10 or top 20% referring to US News and World Report rankings, Bryan’s advice is to clarify what is it about the top 10 schools or the top 10 programs that you want. Be real honest, at least in your own head with what that thing is. For some people, it’s just about the name and the prestige, sort of an instant CV.

In terms of opportunity, it may not matter that much, depending on your career goals. If your goal is to be an internist or emergency medicine doctor and actually treat patients, where you go to school is not going to matter very much.

On the other hand, if the only thing that makes you want to go into medicine is to be the chair of Otolaryngology at an Ivy League institution, you better start working on your CV now and go to the most prestigious school because it’s going to matter. You are going to have to keep playing that social climbing game for the rest of your life.

[36:41] Final Words of Wisdom

Be honest with yourself.

It’s good to start that early and have a realistic idea about what your career goals are and where they are going to be served.

For many things, rankings are not very relevant.

There is nothing in the rankings that really measures the educational inputs that are going into your training. That is something that you should consider.

These days, most medical students use or rely extensively on third-party resources, and not so much on the faculty. But your clinical rotations and the resources of the hospital, all of that is highly variable. And none of that is captured by US News and World Report. But again, depending on your career goals, and depending on what you want out of medical school, those things may be highly relevant.