Pro/Con: ‘Unnecessary’ Testing

5 Comments

Part of a series. Click here to read Night Shift: Throwing Stones.

In March, the Journal of Academic Emergency Medicine published a study entitled ‘Emergency Physician Perceptions of Medically Unnecessary Advanced Diagnostic Imaging’. The study, authored by Kanzaria, Hoffman, Probst, Caloyeras, Berry & Brook, concluded that “many EPs believe a substantial proportion of [advanced imaging]studies, including some they personally order, are medically unnecessary.” This garnered headlines in the lay press, as well as a blog post on EPM’s site, which prompted a response from Jerry Hoffman, one of the study authors. Hoffman asserted that the study’s conclusions had been “slightly distorted” in the mainstream media. 

To address the concerns, and potential distortion, we asked EPM senior editor William Sullivan, DO, JD and editor-in-chief Judith Tintinalli, MD, to explain their concerns with the way this study characterizes “unnecessary” testing. We then asked study co-authors Jerome Hoffman, MD, Hemal Kanzaria, MD, and Marc Probst, MD to respond.


ADVERTISEMENT

———————————


Who DefinesWho Defines “Unnecessary?”

Many of the discussion points in Dr. Hoffman’s study are well taken. Perhaps emergency physicians do order too many diagnostic tests. Perhaps EPs do feel pressured to order unnecessary studies. Perhaps the liability for missing a diagnosis is less than actually perceived. Shared decision making is undoubtedly one way that US physicians can mitigate diagnostic uncertainties. None of these points were the media focus on Dr. Hoffman’s study, though. Instead, headlines loudly declared how this study showed that 97% of emergency physicians order “unnecessary” testing, a headline that gave all emergency physicians a black eye.


ADVERTISEMENT

In law, statutory ambiguities can mean the difference between freedom and incarceration. In fact, the Supreme Court has ruled that statutes are unenforceable if they are not explicit in their meaning – the so-called “void for vagueness” doctrine. We should use this same level of scrutiny when reading scientific studies.

Dr. Hoffman’s study began with an ambiguous question when it asked physicians how much “unnecessary” testing they performed. At one point the authors define an “unnecessary test” as one that “cannot be expected to change the pretest probability of disease in a clinically meaningful way,” yet the study’s questionnaire defined an “unnecessary test” as “a study you would not order if you had no external pressures and were only concerned with providing optimal medical care.” These definitions aren’t remotely equivalent. The former definition doesn’t take into account testing needed for treatment and would make most orthopedic x-rays “unnecessary” since, for example, they don’t “change the pre-test probability” that a deformed wrist is broken. The latter definition uses vague language. What are “external pressures”? Does “optimal care” mean “best possible” care or “cost effective” care? No one knows. Leaving an essential term in the study ambiguously defined tends to invalidate many of the study findings. If every study participant has a different perception of “unnecessary,” how can the study authors purport to reliably compare and tabulate the results? Without a clear definition, the “97% unnecessary tests” sound bite becomes an issue of which semantics will generate the most media buzz.

Dr. Hoffman’s study also prompts a desired response from study participants. Two pages of the study underline the term “unnecessary,” many of the questions focus on the degree of “unnecessary” testing rather than first asking if unnecessary testing even exists, and follow up questions seek input on reducing unnecessary testing – already assuming that “unnecessary” testing does indeed occur. Of the hundreds of checkboxes on the survey, not one allows respondents to state that they do not believe “unnecessary testing” occurs in the emergency department.

Finally, in addition to assuming that ill-defined “unnecessary” care exists, the study is at least partially based on another dubious premise, namely that the increase in advanced imaging “hasn’t improved outcomes” and has led to “overdiagnosis,” “overtreatment,” and “substantial harm.” The citations supporting these allegations are less than robust.


ADVERTISEMENT

When CMS proposed using outcome-based normal CT scans in non-traumatic headaches (one of the potentially “unnecessary” tests cited in Dr. Hoffman’s study) as part of its Quality Data Reporting Program, ACEP objected, noting that the measure did not follow established guidelines and that using the measure would result in missing diagnoses.

The incidence of “unnecessary” testing in the emergency department was grossly overestimated by the shortcomings in this study. In reality, such testing likely amounts to budget dust in our health care spending. Healthcare providers can be held civilly and even criminally liable if they bill for “unnecessary” testing, so “unnecessary” is a definition that we cannot use flippantly. While shared decision making should be incorporated into many physician/patient encounters, equating a physician’s medical decision to obtain advanced imaging as some shortcoming of emergency medical care unnecessarily belittles the work that we do.

Oops. There’s that word again.

———————————

Dont MissDon’t Miss the Point

We suspect most readers are aware of the recent vast increase in use of advanced imaging, which in many cases has not been associated with benefit to patients. This is ubiquitous across medicine, and certainly not unique to emergency medicine. Many well-known initiatives (e.g. Choosing Wisely, “Less is More” sections in major journals) are now targeting the “over $200 billion spent annually on unnecessary services” identified by the Institute of Medicine. Because of false positives, incidentalomas, and over-diagnosis, testing that doesn’t benefit patients actually leads to substantial harm.

We surveyed emergency physicians (EPs) to help understand causes of, and potential solutions to, over-imaging. We ourselves didn’t believe it’s because doctors are ignorant about imaging, or incapable of good judgment. And our survey respondents, by acknowledging that they personally occasionally order a test despite considering it “medically unnecessary” – which we specifically defined as one “you would not order if you had no external pressures and were only concerned with providing optimal medical care” – make it clear that we need to look elsewhere for the key drivers of this behavior.

Not surprisingly, some media outlets employed sensational sound-bites despite our stressing that the take-home message is that physicians face massive pressure from not only a deeply flawed medical-legal system, but also the insane cultural belief (within and outside medicine) that we “must never miss” anything. We obviously could not control everything written by every reporter; we find it ironic, then, that Dr. Sullivan complains about some of this publicity … but goes on routinely to misrepresent our study himself.

We believe it is critical to challenge a culture that does not tolerate uncertainty, and that seeks to punish any bad outcome as “error.” We urge you to read our full journal article, which is in no way negative about Emergency Medicine; in fact, we believe the EPs who participated did us all proud by refusing to pretend these problems don’t exist.

While we prefer to discuss the study itself, we feel obliged to correct some of the inaccuracies in Dr. Sullivan’s essay. For example, we never offered participants any definition of “medically unnecessary” other than that cited above. We also by no means forced respondents to agree that over-testing exists; they had the explicit chance to express that “0% of testing is medically unnecessary” (as was done by 10 of 435 respondents). We also explicitly listed a series of possible external pressures, including patient expectations/requests, fear of malpractice or of missing a diagnosis, and time pressure. Respondents could also write in others, and many described poor patient access to outpatient services, and wishes of other clinicians, as added contributors. These are real pressures we all face in practice; to belittle them or suggest they are irrelevant because of ambiguity is not only inaccurate, but also misses the point of our study.

Dr. Sullivan suggests respondents overstated the amount of unnecessary testing due to a “desired response.” “Social desirability bias” does indeed occur in all survey research – but in this case it most likely led to underestimation of behaviors (“unnecessary” testing) considered undesirable.

We have never suggested that 97% of tests are unnecessary; we did report that 97% of respondents acknowledged personally ordering some (>0%) studies meeting the above definition. We have always noted that this doesn’t mean 97% of all EPs would make the same claim … but also that the precise percentage is irrelevant; the near universal endorsement by survey participants clearly suggests that it is at least a common belief among EPs.

Importantly, this is not a problem of “bad doctors.” Surveyed EPs made it clear that over-testing – which is detrimental to our patients, our profession, and society at large – is not caused by ignorance, or poor medical judgment; thus it cannot be fixed by scolding or “educating” physicians. That so many acknowledged ordering some tests for non-medical reasons means we need to address the system and culture in which we practice. But we can only begin to do this if we first acknowledge the enormous pressures we face in everyday practice; pretending they don’t exist is the last thing we need.

———————————

On the Power of Words
By William Sullivan, DO, JD & Judith Tintinalli, MD

William Sullivan, DO, JD: George Carlin once joked that anyone who drives slower than you is an idiot and anyone who drives faster than you is a maniac. Substitute ordering of medical tests for driving speed and suddenly the subjectivity of the paradigm isn’t quite so amusing – especially when another random person is the yardstick by which your medical care is being judged. The authors seem to miss the point of this critique. There is no argument that their survey shows motives for ordering tests or that it provides insights into external pressures imposed upon emergency physicians. However, contrary to the authors’ assertions, nowhere do we state that they suggested 97% of tests are unnecessary, that respondents were “forced” to agree that over-testing exists, that the study was about “bad doctors,” or that external pressures on emergency physicians don’t exist. These are all straw man arguments. Instead, we critiqued the study semantics and showed how the study overemphasized vaguely defined notions of “unnecessary testing” and “optimal medical care.” The survey wording was clearly slanted to obtain the answers the authors wanted. The script used in the study even notes that “unnecessary tests” may still be considered “appropriate” and may have “some potential value.” What??? Then the authors attempt to generalize the results of a convenience sample of respondents who guessed how much “unnecessary” testing their colleagues performed. Rather than responding to these critiques, the authors take offense and inaccurately accuse us of “misrepresenting” their study. The authors assert that advanced imaging doesn’t benefit patients “in many cases,” but ignore instances in which advanced imaging saves lives. The authors also ignore the benefits of advanced imaging from a patient’s perspective – such as the peace of mind from knowing a serious disease is not present. This study’s imprecise definitions and biased rubric fueled a negative media hype about emergency medical care – as evidenced by multiple articles highly-ranked in search engines that linked their study and had negative connotations toward emergency medicine. This is the issue. The implications of the words we use and the perceptions they create can be significant. Just ask George Carlin fans.

Judith Tintinalli, MD: It takes guts to be an emergency physician, and we do the best we can especially in clouded circumstances. That’s why using terms like ‘medically unnecessary testing’ is the worst use of words I can imagine: vague, contestable, argumentative, and maybe even imaginary.

ABOUT THE AUTHORS

SENIOR EDITOR
Dr. Sullivan, an emergency physician and clinical assistant professor at two residency programs in Illinois, is EPM's resident legal expert. As an attorney specializing in healthcare issues, Dr. Sullivan represents physicians and has published many articles on legal aspects of medicine. He is a past president of the Illinois College of Emergency Physicians and a past chair and current member of the American College of Emergency Physicians' Medical Legal Committee.

EDITOR-IN-CHIEF Dr. Tintinalli is currently a professor and Chair Emeritus of Emergency Medicine at the University of North Carolina. In addition to teaching in the emergency medicine department, she is an adjunct professor at the UNC Gillings School of Global Public, and a frequent lecturer in the School of Journalism and Mass Communication. Dr. Tintinalli is double boarded in emergency medicine and internal medicine. She was the founder and first president of the Council of Emergency Medicine Residency Directors. She is a former president of ABEM as well as the Association of Academic Chairs in Emergency Medicine. She is a past winner of ACEP's James Mills award as well as ACEP's National Education Award. And of course, she is the Editor-in-Chief of 7 editions of her eponymous textbook, which is arguably the best-known EM text in the world.

Jerome Hoffman, MD is Professor Emeritus of Medicine at the UCLA School of Medicine.

Dr. Kanzaria is an emergency physician and Clinical Scholar affiliated with the US Department of Veterans Affairs and the University of California Los Angeles.

Dr. Probst is an Assistant Professor and K23 Scholar in the Department of Emergency Medicine at the Icahn School of Medicine at Mount Sinai.

5 Comments

  1. Like Wheeler’s thought experiments there seems to be a certain quantum nature to “unnecessary” tests. A seemingly unnecessary low probability test has a quantum eraser applied to it and becomes necessary whenever you end up saying to yourself, “crap, I’m glad I ordered that test after all.”

  2. Roger Perry, MD, PhD on

    The study “Emergency Physician Perceptions of Medically Unnecessary Advanced Diagnostic Imaging”, was a poorly designed and implemented study from the start with a host of flaws, not the least of which was the bias of at least one of the author, Jerome Hoffman. I agree with the criticisms of Dr. Sullivan and Dr. Tintinalli, who highlighted some of the flaws in the study. The research questions were constructed to obtain support for the preconceived opinion of the authors, that emergency physicians order unnecessary tests. In addition, I would like to highlight one of the aspects missing from the study queries and that is the omission of assessment of the number and kinds of unanticipated findings on the tests ordered. A final comment, who ever paid for this flimsy and prejudice study should request a refund.

  3. The problem with the AAEM study on unnecessary advanced diagnostic imaging in the ED is that it is like the first incandescent light bulbs: too much heat and not enough light. Aside from all the issues in the study with definitions of unnecessary testing and vague language and the potential for media ‘misrepresentation’; the study asked the wrong question, and failed to shed much light on what are far more important considerations related to cost-effective care. Let me use an example. CT scans of the abdomen have been well recognized as a valuable diagnostic tool in patients with acute flank pain and hematuria, and the use of these scans in these patients in the ED has increased substantially. With this increase, the likelihood that some of these scans were ‘unnecessary’ likewise goes up, to the point where Choosing Wisely has incorporated strategies meant to address this opportunity to reduce the cost of care.

    The question of whether 97% of emergency physicians agree that some of the CT scans performed in the ED were unnecessary, or failed to change the patient’s management, or were not useful even though they ‘may still be considered appropriate and have some potential’ (insert your favorite synonym), or if just a few ACEP Cost Effective Care Task Force members agree: this is really irrelevant.

    If you assume that at least some of these CTs do not need to be done, the type of questions that really deserve answers are ones like these:
    1. Can we eliminate (or redirect to a more affordable venue) some of these CTs and maintain (or improve) quality and outcomes? How many patients would quality?
    2. When can we do this without increasing liability risk?
    3. Who would reap the financial benefits of such a strategy (patients, providers, payers, hospitals), and is the benefit worth the effort?
    4. How much do alternative studies like ultrasound enhance the CT use-reduction strategy?
    5. What are the characteristics of EPs (certification, malpractice experience, practice experience, financial incentives) that allow them to adopt the strategy more or less successfully?

    Of course, the media could misrepresent any of these questions and studies; but the difference is that the answers could really help us respond to the cost-effective care challenge EM faces. The AAEM survey / study really doesn’t shed much light on the issues, other than to confirm what is mostly obvious (as surveys are wont to do). If so many emergency physicians already agree to the premise, it’s pretty much a given, and who is left to convince?

  4. Tom Benzoni on

    Ummmmmmmmmm.
    Instead of casting aspersions, could we discuss the topic?
    It appears both groups succumbed to the “see this fist? missed that one!” method.
    Works every time with docs!
    I think, with the brain trust above, we should get a better discussion of the topic, like how over-testing adds to liability, not subtracts, by making a CT of the c-spine “routine of care” whether or not the patient has a likely lesion discoverable by CT.
    Or how we may be ordering more CT in trauma without discovering more actionable lesions.
    Let’s stay on topic.
    We have enough people sniping at us without forming a circular firing squad.

  5. Stop! There are truths in both arguments. We all know ED physicians who, either from fear of litigation or insecurity in their clinical judgement, order too many tests. We joke about the doc who’ll “test until they find something – anything” or the doc who “If they D/C you from the ER, you’re guaranteed to live to 100”. We also know internists, surgeons, etc. who do the same. On the other hand, working in the ED for almost 30 years, I know that most ER docs really do their best to do the right thing for their patient. This issue isn’t worth fighting about – we need to be courageous enough to look at it and make decisions based on evidence. I agree that this “study” was subjective and based on perception, I didn’t take it at all to be a slam on ED physicians.

Leave A Reply