Pro/Con: ‘Unnecessary’ Testing

Part of a series. Click here to read Night Shift: Throwing Stones.

In March, the Journal of Academic Emergency Medicine published a study entitled ‘Emergency Physician Perceptions of Medically Unnecessary Advanced Diagnostic Imaging’. The study, authored by Kanzaria, Hoffman, Probst, Caloyeras, Berry & Brook, concluded that “many EPs believe a substantial proportion of [advanced imaging]studies, including some they personally order, are medically unnecessary.” This garnered headlines in the lay press, as well as a blog post on EPM’s site, which prompted a response from Jerry Hoffman, one of the study authors. Hoffman asserted that the study’s conclusions had been “slightly distorted” in the mainstream media. 

To address the concerns, and potential distortion, we asked EPM senior editor William Sullivan, DO, JD and editor-in-chief Judith Tintinalli, MD, to explain their concerns with the way this study characterizes “unnecessary” testing. We then asked study co-authors Jerome Hoffman, MD, Hemal Kanzaria, MD, and Marc Probst, MD to respond.

ADVERTISEMENT

———————————

Who Defines “Unnecessary?”

Many of the discussion points in Dr. Hoffman’s study are well taken. Perhaps emergency physicians do order too many diagnostic tests. Perhaps EPs do feel pressured to order unnecessary studies. Perhaps the liability for missing a diagnosis is less than actually perceived. Shared decision making is undoubtedly one way that US physicians can mitigate diagnostic uncertainties. None of these points were the media focus on Dr. Hoffman’s study, though. Instead, headlines loudly declared how this study showed that 97% of emergency physicians order “unnecessary” testing, a headline that gave all emergency physicians a black eye.

ADVERTISEMENT

In law, statutory ambiguities can mean the difference between freedom and incarceration. In fact, the Supreme Court has ruled that statutes are unenforceable if they are not explicit in their meaning – the so-called “void for vagueness” doctrine. We should use this same level of scrutiny when reading scientific studies.

Dr. Hoffman’s study began with an ambiguous question when it asked physicians how much “unnecessary” testing they performed. At one point the authors define an “unnecessary test” as one that “cannot be expected to change the pretest probability of disease in a clinically meaningful way,” yet the study’s questionnaire defined an “unnecessary test” as “a study you would not order if you had no external pressures and were only concerned with providing optimal medical care.” These definitions aren’t remotely equivalent. The former definition doesn’t take into account testing needed for treatment and would make most orthopedic x-rays “unnecessary” since, for example, they don’t “change the pre-test probability” that a deformed wrist is broken. The latter definition uses vague language. What are “external pressures”? Does “optimal care” mean “best possible” care or “cost effective” care? No one knows. Leaving an essential term in the study ambiguously defined tends to invalidate many of the study findings. If every study participant has a different perception of “unnecessary,” how can the study authors purport to reliably compare and tabulate the results? Without a clear definition, the “97% unnecessary tests” sound bite becomes an issue of which semantics will generate the most media buzz.

Dr. Hoffman’s study also prompts a desired response from study participants. Two pages of the study underline the term “unnecessary,” many of the questions focus on the degree of “unnecessary” testing rather than first asking if unnecessary testing even exists, and follow up questions seek input on reducing unnecessary testing – already assuming that “unnecessary” testing does indeed occur. Of the hundreds of checkboxes on the survey, not one allows respondents to state that they do not believe “unnecessary testing” occurs in the emergency department.

ADVERTISEMENT

Finally, in addition to assuming that ill-defined “unnecessary” care exists, the study is at least partially based on another dubious premise, namely that the increase in advanced imaging “hasn’t improved outcomes” and has led to “overdiagnosis,” “overtreatment,” and “substantial harm.” The citations supporting these allegations are less than robust.

When CMS proposed using outcome-based normal CT scans in non-traumatic headaches (one of the potentially “unnecessary” tests cited in Dr. Hoffman’s study) as part of its Quality Data Reporting Program, ACEP objected, noting that the measure did not follow established guidelines and that using the measure would result in missing diagnoses.

The incidence of “unnecessary” testing in the emergency department was grossly overestimated by the shortcomings in this study. In reality, such testing likely amounts to budget dust in our health care spending. Healthcare providers can be held civilly and even criminally liable if they bill for “unnecessary” testing, so “unnecessary” is a definition that we cannot use flippantly. While shared decision making should be incorporated into many physician/patient encounters, equating a physician’s medical decision to obtain advanced imaging as some shortcoming of emergency medical care unnecessarily belittles the work that we do.

Oops. There’s that word again.

ADVERTISEMENT

———————————

Don’t Miss the Point

We suspect most readers are aware of the recent vast increase in use of advanced imaging, which in many cases has not been associated with benefit to patients. This is ubiquitous across medicine, and certainly not unique to emergency medicine. Many well-known initiatives (e.g. Choosing Wisely, “Less is More” sections in major journals) are now targeting the “over $200 billion spent annually on unnecessary services” identified by the Institute of Medicine. Because of false positives, incidentalomas, and over-diagnosis, testing that doesn’t benefit patients actually leads to substantial harm.

We surveyed emergency physicians (EPs) to help understand causes of, and potential solutions to, over-imaging. We ourselves didn’t believe it’s because doctors are ignorant about imaging, or incapable of good judgment. And our survey respondents, by acknowledging that they personally occasionally order a test despite considering it “medically unnecessary” – which we specifically defined as one “you would not order if you had no external pressures and were only concerned with providing optimal medical care” – make it clear that we need to look elsewhere for the key drivers of this behavior.

Not surprisingly, some media outlets employed sensational sound-bites despite our stressing that the take-home message is that physicians face massive pressure from not only a deeply flawed medical-legal system, but also the insane cultural belief (within and outside medicine) that we “must never miss” anything. We obviously could not control everything written by every reporter; we find it ironic, then, that Dr. Sullivan complains about some of this publicity … but goes on routinely to misrepresent our study himself.

We believe it is critical to challenge a culture that does not tolerate uncertainty, and that seeks to punish any bad outcome as “error.” We urge you to read our full journal article, which is in no way negative about Emergency Medicine; in fact, we believe the EPs who participated did us all proud by refusing to pretend these problems don’t exist.

While we prefer to discuss the study itself, we feel obliged to correct some of the inaccuracies in Dr. Sullivan’s essay. For example, we never offered participants any definition of “medically unnecessary” other than that cited above. We also by no means forced respondents to agree that over-testing exists; they had the explicit chance to express that “0% of testing is medically unnecessary” (as was done by 10 of 435 respondents). We also explicitly listed a series of possible external pressures, including patient expectations/requests, fear of malpractice or of missing a diagnosis, and time pressure. Respondents could also write in others, and many described poor patient access to outpatient services, and wishes of other clinicians, as added contributors. These are real pressures we all face in practice; to belittle them or suggest they are irrelevant because of ambiguity is not only inaccurate, but also misses the point of our study.

Dr. Sullivan suggests respondents overstated the amount of unnecessary testing due to a “desired response.” “Social desirability bias” does indeed occur in all survey research – but in this case it most likely led to underestimation of behaviors (“unnecessary” testing) considered undesirable.

We have never suggested that 97% of tests are unnecessary; we did report that 97% of respondents acknowledged personally ordering some (>0%) studies meeting the above definition. We have always noted that this doesn’t mean 97% of all EPs would make the same claim … but also that the precise percentage is irrelevant; the near universal endorsement by survey participants clearly suggests that it is at least a common belief among EPs.

Importantly, this is not a problem of “bad doctors.” Surveyed EPs made it clear that over-testing – which is detrimental to our patients, our profession, and society at large – is not caused by ignorance, or poor medical judgment; thus it cannot be fixed by scolding or “educating” physicians. That so many acknowledged ordering some tests for non-medical reasons means we need to address the system and culture in which we practice. But we can only begin to do this if we first acknowledge the enormous pressures we face in everyday practice; pretending they don’t exist is the last thing we need.

———————————

On the Power of Words
By William Sullivan, DO, JD & Judith Tintinalli, MD

William Sullivan, DO, JD: George Carlin once joked that anyone who drives slower than you is an idiot and anyone who drives faster than you is a maniac. Substitute ordering of medical tests for driving speed and suddenly the subjectivity of the paradigm isn’t quite so amusing – especially when another random person is the yardstick by which your medical care is being judged. The authors seem to miss the point of this critique. There is no argument that their survey shows motives for ordering tests or that it provides insights into external pressures imposed upon emergency physicians. However, contrary to the authors’ assertions, nowhere do we state that they suggested 97% of tests are unnecessary, that respondents were “forced” to agree that over-testing exists, that the study was about “bad doctors,” or that external pressures on emergency physicians don’t exist. These are all straw man arguments. Instead, we critiqued the study semantics and showed how the study overemphasized vaguely defined notions of “unnecessary testing” and “optimal medical care.” The survey wording was clearly slanted to obtain the answers the authors wanted. The script used in the study even notes that “unnecessary tests” may still be considered “appropriate” and may have “some potential value.” What??? Then the authors attempt to generalize the results of a convenience sample of respondents who guessed how much “unnecessary” testing their colleagues performed. Rather than responding to these critiques, the authors take offense and inaccurately accuse us of “misrepresenting” their study. The authors assert that advanced imaging doesn’t benefit patients “in many cases,” but ignore instances in which advanced imaging saves lives. The authors also ignore the benefits of advanced imaging from a patient’s perspective – such as the peace of mind from knowing a serious disease is not present. This study’s imprecise definitions and biased rubric fueled a negative media hype about emergency medical care – as evidenced by multiple articles highly-ranked in search engines that linked their study and had negative connotations toward emergency medicine. This is the issue. The implications of the words we use and the perceptions they create can be significant. Just ask George Carlin fans.

Judith Tintinalli, MD: It takes guts to be an emergency physician, and we do the best we can especially in clouded circumstances. That’s why using terms like ‘medically unnecessary testing’ is the worst use of words I can imagine: vague, contestable, argumentative, and maybe even imaginary.