USMLE Step 1 Went Pass/Fail: What Does That Mean for Premeds?

Session 379

What are residency program directors looking for in applicants? Is the USMLE Step 1 the most important factor? We’re looking at the data and the answer might surprise you!

If you’re a conspiracy theorist, a doomsday or a prepper or any other type of person who thinks that change is bad, this may be a good episode for you.

Be sure to check out everything else on the Meded Media.

Listen to this podcast episode with the player above, or keep reading for the highlights and takeaway points.

[00:47] What Board Directors Think of USMLE’s Pass/Fail System

The United States Medical Licensing Examination, the USMLE, which is created by the Federation of State Medical Boards and the National Board of Medical Examiners announced that it was changing the first part of their exam, which is the Step 1 to pass/fail.

What previously was a three-digit score that was often used to determine your strength as a candidate to very competitive specialties is no longer going to be a three-digit score.

If you haven’t yet, check out our Specialty Stories Podcast for a better understanding of the different specialties out there. Some of the physicians I’ve talked with on that podcast also shared their points of view on the USMLE Step 1 exam.

I specifically asked them what was going to change. And a lot of them actually didn’t know or that they haven’t figured it out yet.

And some of them said it’s not going to change much because they review every application and that’s just the way it is.

“Residencies are not getting tens of thousands of applications per residency.”Click To Tweet

Remember that this is not medical school. Medical schools are getting 10,000 applications. They’re getting thousands upon thousands upon thousands of applications and they have to filter through those quickly.

Residencies are not getting that many applications per program. That’s the first thing you have to keep in mind is that if a program director will review every application that’s much more believable than medical school.

And if a program is going to review every application, then they take the time to review things a little bit deeper.

[03:40] Embrace Change!

The whole process – of going to medical school, getting licensed, becoming a physician, applying for residency and getting residencies – all of that has constantly been changing. And that will always constantly change.

“This is just one of those changes. The world is not going to end.”Click To Tweet

A lot of people are screaming that this change is going to hurt a lot of people, the Caribbean graduates, or the DOs, or the international medical graduates, etc. This is going to hurt everyone and anybody who isn’t at a top 20 medical school.

That’s what everyone is screaming right now that your school prestige is all of a sudden going to matter the most out of everything. But that just can’t be true.

[05:53] Looking at the NRMP Program Director Survey

Let’s look at the 2018 NRMP program director survey where they surveyed 1,233 program directors.

According to the figure, it says that Step 1 was the most cited factor in determining their wanting to invite you for an interview. So this isn’t ranked from best to worst.

This figure is just saying the majority, far majority. And they ranked it a 4.1 which is the most important number here.

“94% of all of the 1,233 program directors said they think of Step 1 and Level 1 as a factor in determining who they want to interview.”Click To Tweet

The second thing on this list is letters of recommendations in the specialty. When you’re applying to medical school you have to go get letters of recommendations from your professors and doctors and supervisors and whoever else.

86% of the 1,233 program directors said that letters of recommendation are a factor in determining who they want to invite for an interview.

Again, it’s not the 86% most important factor, it’s just 86% of the program directors said it was a factor. And it’s rated at 4.2. Hence, the most important is actually letters recommendation because it’s rated at a 4.2.

“You have to understand how to review data and look at these surveys too to really understand what's going on in the world.”Click To Tweet

So everyone’s screaming that step one is the most important factor. Well maybe, but we don’t have any data that shows that. What we have is data that shows that it is the most cited factor in determining who to invite for an interview with a rating of 4.1.

[09:26] Other Rating Factors

The dean’s letter is something that all medical students get from their school dean laying out who you are. It’s like a committee letter from your dean. In premed terms, this is a letter that kind of aggregates all the information. So it makes it easier for the program directors to get a glance of who you are.

Step 2 CK (Clinical Knowledge) is listed on here. Only 80% of the program directors say they consider it a factor at a 4.0. So not far behind the 4.1 Hence, it’s almost just as important, just that less program directors actually look at it.

Personal statements are at 78% but only ranked 3.7. So personal statements really maybe aren’t that important.

Awards or special honors in basic sciences got the lowest score here at 3.3. Everyone gets a trophy these days.

“If you have failed Step 1 or Level 1, that's a huge red flag and it’s a potential warning to them that maybe you're a bad test taker.”Click To Tweet

I’ve heard this a bunch on the Specialty Stories when I interview the program directors, they look very carefully at Step 1/ Level 1 scores as a huge red flag. This gives them an idea that you’re probably a bad test taker. Maybe you have some anxiety around tests, whatever that is.

But the testing is not over once you’re done with USMLE. You have your actual specialty boards. And failing this is bad. This is ranked at 4.5

The only thing higher than 4.5 is if you were actually flagged with a violation in the match program. That’s obviously a very important rating at 4.8.

Your class ranking quartile is 70%. This is interesting because 70% is a lot. It’s not as much as 94% for step one. 70% of the program directors cited your class ranking quartile as a factor just like step one is a factor. They rated it as a 3.9 so it’s kind of important, although not as important as Step 1, which is at 4.1 as well as letters of recommendation at 4.2.

Perceived commitment to specialty is 4.3. Obviously, they want to know that you really want to do this right.

[14:00] Commitment to Specialty, Audition Elective, AOA, and More!

“Prior knowledge of the applicant is not what you know but it's who you know that will stand the test of time, always.”Click To Tweet

If the program director knows you then you have a better shot of matching at that program. Hopefully if they like you obviously, but you have a better shot of matching that program because they know you.

So if you want to go to a program, get them to know you and do well on your clerkship. This is what’s being repeated over and over and over again in my Specialty Stories podcast when I interview program directors.

Audition elective is 4.2. This is rotation within your department. 65% of program directors are citing it as a factor. Doing the audition elective is evidence of professionalism and ethics. But only 65% of program directors are stating it as a factor. The other 35%, they don’t care.

Leadership qualities, 61% and 4.1 while AOA is 60% and 3.9. AOA or Alpha Omega Alpha is an honors society. It’s just recognition for doing well in your preclinical years.

Life experience is 3.8 so it’s not that high. Perceived interest in the program is 4.1. You want to go to that program, not just because it’s there, but you actually want to go to that program specifically.

Step 2 CS (Clinical Skills) is 4.2. This is where you actually go and you examine a patient. It’s kind of the standardized patient world. 4.2 so you’ve got to pass this because it’s important for them.

[16:16] Is Where You Went to Medical School Still a Factor?

Only 50% of program directors said that is a factor in determining who they will invite for an interview. And they only ranked it at a 3.8 which isn’t very high in this chart.

That being said, do you think removing something like Step 1 is going to move something that’s way down on the list all the way to the top?

Will some programs look at where you went to medical school a little bit more? Maybe. And you will see this on their website.

“Are the programs who have always looked at where you went to medical school still going to look at where you went to medical school? Yes, because that's what they care about.”Click To Tweet

The programs that care about that will always care about that. And that won’t change the programs who understand that that doesn’t matter in the world, in your practice with you taking care of patients. Those programs aren’t going to care still.

[18:32] Don’t Just Go for the Top 20 Medical Schools

The Top 20 Medical Schools by U.S. News World and Reports is a big flaming pile of poop. It’s all subjective data. Go look at how they actually put together the rankings of medical schools.

Medical schools love them, especially the ones that are ranked at the top. But they don’t mean anything.

So stop trying to only go to a top 20. Don’t get me wrong, shoot for the stars. If you want to go to Columbia because you want to go to Columbia, then go to Columbia. If you want to go to Harvard because Harvard is where you want to be, then go to Harvard.

What I’m saying is don’t go to Harvard because it’s a top five school because top five school doesn’t mean anything because the list is trash.

Prestige is not going to matter so much now. It hasn’t mattered in the past at 3.8. Only 50% of program directors are saying it’s a factor and how they look at applicants. It’s probably not going to change much.

[20:25] Do We Need an Objective Measure?

What a lot of students are saying and where a lot of people are going is that Step 2 is going to take over as the new thing to look at and that makes a lot more sense.

94% of program directors are stating USMLE Step 1 as a factor with a 4.1. 80% are stating Step 2 score at a 4.0 as a factor. It’s much more likely that the average rating is the bigger factor.

The average rating could go up to a 4.1 or 4.2 but I’m saying that more program directors are probably going to cite it as a factor so that’ s very likely.

Personally, I wished that when USMLE changed Step 1 to pass/fail that they also changed all of their tests to pass/fail.

The medical licensing exam is a credentialing exam for you to get a medical license. It’s a national test that every state uses to determine whether or not you are eligible to get a medical license. And that doesn’t need a score. It just needs a yes or a no.

How we have turned this into a determination of your ability to be a radiologist, an orthopedic surgeon or anything else has completely bastardized the whole system.

It’s easier for program directors to look at a number. But we don’t need an objective measure. When you are done with your residency and you are applying for a job, what objective measure is there? They don’t look at your Step scores. Instead, they look at your medical license and whether you’ve been dinged or sued.

“We don't have test scores as objective measures in life so program directors don't need an objective measure to determine who they want to interview for an applicant.”Click To Tweet

So the highest cited factor here is 86% at a 4.2 which is a subjective thing is a letter of recommendation. 86% of the program directors are saying they really want that letter of recommendation 4.2 so that’s subjective. The Dean’s letter is subjective as well as personal statement.

Some may think grades are subjective in clerkships. You’re still taking tests, but some of the grades are very heavily weighted for whoever’s running the clerkship and who your attendings are.

Class ranking is objective. Commitment to specialty is completely subjective. Knowledge of applicant is completely subjective. Life is subjective.

“For those of you screaming that there needs to be an objective measure, oh well life isn't fair. Life isn't built on only objective data.”Click To Tweet

If Step 2 continues to be scored, then it will probably rise in importance because it’s easier. There are program directors out there who are busy and they want an easy button. And they will press that easy button to filter their applicants.

[25:04] How the Scores Affect the Student’s Specialty of Choice

Let’s talk about the students who probably took step one and they were thinking about internal medicine. But with this score, maybe they think they should try radiology. Or with this score, maybe they’re thinking they should try dermatology. It would be a great study to go back and look at physicians who are burnt out.

It’s interesting to know if their Step 1 score influenced their choice of specialty. Did they go into a more competitive specialty because they crushed Step 1? I would love to see data on that.

How many times have we heard from someone saying, well, I wanted to go into the family practice. But I was told that’s stupid because I scored 99th percentile on my Step1. I should go into something else.

Maybe this will allow students to go follow their heart more than what their score is telling them to follow.

[26:58] IMGs and DOs

Until things change, we won’t know until we have data. We won’t know. Will this have the potential to affect students at a DO school or at a Caribbean school or any other international medical schools? At this point, we still don’t know.

86% of program directors at 4.2 rating said the letter of recommendation in the specialty was important. Your Dean’s letter is important. Step 2 or Level 2 score is important. Personal statements are important. None of that says “only from allopathic school.”

“Nobody says it should be a letter of recommendation in the specialty only from allopathic students. This is a letter of recommendation in the specialty – period.”Click To Tweet

It doesn’t matter if you’re at an MD school at a DO school or Caribbean or other international medical school.

But that being said, I’m not diminishing the fact that Step 1 has historically been used to potentially up the game of international medical graduates who have struggled previously.

If you started poorly in undergrad, if you did terrible on the MCAT because you just did and you are at a Caribbean school, Step 1 was your cheat code to get ahead in the world. It would potentially make you a much more competitive applicant for some of the more competitive specialties.

If that’s what you wanted and that may hurt you, Step 2 may be right there behind you to help you out.

Step 2 is right there still as a scored test still there for you to do well on that can potentially still help you. Again, letters of recommendations, Dean’s letter, step two, score, personal statement, grades and clerkships.

All those are still important no matter what school you go to.

If you think with Step 1 going pass fail that it’s going to completely derail your ability to practice medicine in the future, you are wrong.

It may be a little bit trickier. There may be some different things you may need to think of now in the future, you may need to do a little bit more work, building relationships, shining a little bit more during your rotations so that you can get some good grades from those clerkship directors, and attendings.

Build those relationships so you get an even better letter of recommendation. All of that has always been important and maybe more important in the future. And so will this potentially hurt the introvert who doesn’t like to stand out during clerkships, which potentially leads to weaker letters of recommendations.

[31:40] The Fate of Step 1

This is just a change and more will change. Understand that this is not done to make it harder for you. Students will always find something to complain about. This is a big change, but it’s a change and things have always changed.

“With every change, just remember to take a deep breath to understand that the world is not ending.”Click To Tweet

Think about the students in the early nineties who were going to school and all of a sudden there’s this new USMLE thing. Think about those students. It’s just a change. Those students survived.

You will survive if you’ve listened to the show long enough. You know that I am pretty optimistic about most things and some people see it as a fault.

Until we understand it more, we won’t know anything. But all I do know is that everything in life is determined by the effort that you put into it.

You will have to put in more effort potentially to build those relationships to do better on Step 2 and to do better in your clerkships. All of anything you get in this world is through your effort.

So continue to work hard, continue to push hard, and be excited.

That’s Step 1 – the test that students freaked out about that they thought would determine the rest of their life is no longer there. Let’s hope that Step 2 goes that same way.