How important are in-service exam scores when considering a new hire?
Dear Director,
I’m hiring a new physician and I really like a certain senior resident but their residency director told me they failed their in-service exam. Everything else about this candidate seems perfect. Am I setting myself up for trouble if I bring someone on who may be academically challenged?
There are many traits that make up a successful physician, test taking ability being just one of them. While low test scores could be the result of a critical lack of knowledge, a great in-training exam score doesn’t guarantee success as an attending physician either. The doc who has the textbook memorized may have trouble interacting with patients and staff, or may have trouble making a definitive clinical decision. Bottom line: Find out more of this applicant’s story before you make a decision.
Let’s start by differentiating the in-training exams from the written boards. While I like to see people do well on in-training exams, particularly in their last 2 years of residency, I’m not sure it’s fair to judge someone by a performance 15+ months before they complete their training. As ABEM states on their website, the in-training exam targets the knowledge base that is expected of an EM3 resident months before their graduation. Although residencies may equate a passing score to it, ABEM does not hold out a set score but rather allows residency programs to judge an individual resident’s progress towards successful ABEM certification. That said, there is a clear relationship between success on the in-training exam and success on the written board exam, and failing a board exam (particularly repeatedly) is certainly a red flag.
Knowledge Deficits
I’ve worked with some amazingly smart physicians. I’ve also worked with some where I wondered how they got through medical school. When I hire someone out of residency, I assume they have obtained the critical level of knowledge to successfully function in an ED. I expect to coach them on speed, efficiency, core measures and hospital politics, but not on how to recognize a STEMI.
While there is variability in the core knowledge base, there is also a threshold below which is it difficult to be an effective physician. It is very rare for a resident to be below this threshold and successfully complete residency. Still, I’m watching out for those few residents who have fallen through the cracks. Clinically, I’m looking for patterns of competence or incompetence. Most of our day-to-day job is fairly straight forward – you don’t need to be a rocket scientist to provide non-negligent care – but when one of my docs has failed a board exam, I might evaluate their skill set more critically. I use q/a indicators as well as my “gut” judgment when I’m discussing a case during a shift or during sign out with the person.
References
This is a situation where references could make all the difference. While something as significant as a poor performance on an in-training exam needs to be discussed with the applicant, the references will likely be able to put this into perspective. Does your opinion change if you find out the applicant had been up all night on call during an ICU rotation or was on maternity or paternity leave because they had a 1 week old? These things, among others, open my mind to ask the next question, which is how is the applicant in the clinical arena. How do they compare to their residency classmates—top third, average, bottom third? A bad in-training exam on a resident regarded in the bottom third of a residency is probably not someone I want to hire when compared to a sleep-deprived score with a resident who is considered among the top in their class clinically. The reference becomes even more valid if you personally know the reference and have a history of trust.
More than Meets the Eye
I remember working with someone who failed their recertification exam. Failure rate on this is very low so bells went off in my mind as I wondered if there was a clinical issue. I asked what happened. He told me that he scheduled his test late when there were few available spots. He ended up taking the test after a night shift. He did nights regularly and thought he’d be fine. Unfortunately, he got there, fell asleep, and woke up at the end of his timed exam to realize he only completed about a third of the questions. The possibility of passing was zero. He passed the exam on his next opportunity. Sometimes a poor result does not mean a poor understanding of the material. It could be bad luck, or it may represent an active decision not to study for an exam with consideration that a top score is not essential (it’s a pass/fail test) and the person was better served working on other endeavors.
Conclusion
Success as an attending physician goes beyond an in-training exam score. Interviews are used to assess personality, enthusiasm for the job, an applicant’s understanding of the department’s vision and objectives, to name a few. There is an assumed baseline knowledge base of any applicant and it’s up to you, as the medical director, to check references to be sure that those you hire meet your standards. Because these are residents, it’s also worthy to note the circumstances of any particular situation. In the long run, if you can justify the situation with the in-training exam and the physician has references that support their skill as a clinician, I think you can overlook one exam that takes place just past the mid-point of a resident’s training.
What do in-training exams mean anyway?
Unlike other ABEM examinations, the in-training examination does not have a passing score. It is a standardized examination that residents and program faculty can use to judge an individual resident’s progress toward successful ABEM certification. There is a strong relationship between in-training and qualifying examination scores – physicians with higher in-training scores have a higher likelihood of passing the qualifying examination.
