When Evaluating EMR Efficacy, Where’s the Beef?

Why aren’t health information systems subjected to the same rigorous research as are other complicated clinical procedures?

Why aren’t health information systems subjected to the same rigorous research as are other complicated clinical procedures?

In 1984, Wendy’s ran an iconic advertisement in which an ornery, elderly woman is handed a massive burger bun. “Where’s the beef?!” she cried. Since this series of commercials, that phrase has become synonymous with questioning the substance of an idea or product. When we look at evidence in medicine, the “beef” is the randomized controlled trial. Although these trials are subject to many sources of bias, they are, when well done, considered the “gold standard.”

ADVERTISEMENT

Individual randomized controlled trials have involved tens of thousands of patients to determine if drug A is better than drug B when both are known to be similarly effective and nuances of difference are being sought. These studies can cost millions of dollars to conduct and often the results are not what the sponsors hoped they would be.

Randomized controlled trials often involve the assessment of complicated clinical procedures that are both costly and/or risky. In this setting it is even more important to find convincing evidence of the value of the procedure over other treatments.

Fast forward to 2012. We are looking at the most expensive healthcare initiative ever undertaken in the nation’s hospitals and physicians’ offices: the incorporation of clinically oriented health information technology. So, where are the randomized controlled trials? Where’s the beef?

ADVERTISEMENT

An article in the September 18th issue of the Wall Street Journal by Stephen Soumerai and Ross Koppel addresses “Where’s the beef?” when it comes to health information technology (HIT). Here are some of the points made:

In two years, hundreds of thousands of physicians and thousands of hospitals will have their Medicare and Medicaid payments reduced if they have not adopted HIT

The government is expected to pay out tens of billions of dollars in subsidies and incentives related to the adoption of HIT

ADVERTISEMENT

HIT lobbyists claim that these systems would reduce healthcare cost by up to $100 B. annually

Since 2009 a third of health care providers have installed some version of HIT

For a major hospital, initial costs of $150-200 million can be expected — and with implementation, linking and integrating systems, training and data entry the bill can come to $1 billion.

But what about the return on investment? The WSJ authors detail the findings of a recent study analyzing the results of HIT implementation (The Economics of Health Information Technology in Medication Management: A Systematic Review of Economic Evaluations, O’Reilly, D., et al, J Am Inform Assoc 2012;19:423-438). Specifically, the areas of interest were cost-effectiveness analysis, cost-utility analysis and cost-benefit analysis. The review was conducted by authors from McMaster University in Hamilton, Ontario.

ADVERTISEMENT

After reviewing almost 3,600 studies, 31 were found meeting the authors’ inclusion requirements. Because of the heterogeneous nature of the studies, the data could not be synthesized and, instead, a narrative review was conducted. Most of the studies were hospital-based and only five performed a full economic analysis. There were only three large randomized, controlled trials. The WSJ authors note that one showed no saving, another an increased cost of $2,200 per year and the last demonstrated a questionable savings of $22 per patient per year.

The McMaster authors concluded that “the quality of economic literature in this area is poor … given the uncertainty that surrounds the costs and outcomes data, and limited study design, it is difficult to reach any definitive conclusion as to whether the additional costs and benefits represent value for money.”
But the train of clumsy, user-unfriendly and inefficient HIT systems has left the station. The WSJ authors observe that by the time “health-care providers find out that the promise of cost savings are an illusion, it will be too late.” “Having spent hundreds of millions on the technology, they won’t be able to afford to throw it out like a defective toaster.”

Here’s a paper making the case for randomized controlled trials of HIT systems. The authors assert that there are many other complex areas of medicine that have been assessed by RCTs and that the analysis of value of use of HIT systems should be subjected to this level of scrutiny.

THE CASE FOR RANDOMIZED CONTROLLED TRIALS TO ASSESS THE IMPACT OF CLINICAL INFORMATION SYSTEMS
Liu, J.L.Y., et al, J Am Med Inform Assoc 18(2):173, March 1, 2011

Implementation of clinical information systems in the healthcare setting is being encouraged on multiple levels, but they are costly and some studies have suggested that their effects on patient outcomes are not consistently beneficial. The randomized controlled trial (RCT) is considered the standard for the evaluation of drug therapy, but some developers of clinical information systems do not feel that RCTs are an appropriate method of evaluating these systems. These Scottish authors comment on the feasibility and advisability of performance of RCTs to evaluate clinical information systems and promote their evidence-based use. RCTs are clearly indicated in the absence of incontrovertible evidence that the benefits of these systems vastly outweigh their harms and costs. While some argue that these systems are too complex to be subjected to RCTs, these authors note the use of RCTs in the setting of other highly complex interventions and believe that a multi-arm approach would facilitate the determination of the relative impact of each component of the systems on patient outcomes. They further suggest that the RCT is the best method of obtaining unbiased estimates of the effects of clinical information systems, and that the information provided by RCTs can be supplemented by studies using other designs. To those who feel that the rapidly expanding technology involved in clinical information systems is not amenable to performance of a RCT, the authors note that other study types involving other rapidly evolving medical interventions also require prolonged follow-up to definitively establish safety and efficacy. The authors challenge the belief that clinical information systems “do no harm,” and suggest that performance of RCTs should be an integral part of their widespread implementation. 115 references (j.l.y.liu@dundee.ac.uk – no reprints)

Copyright 2011 by Emergency Medical Abstracts – All Rights Reserved 8/11 – #14

The adoption of HIT systems has broad implications in the setting of healthcare. Physicians and hospitals purchasing these systems, theoretically at least, should expect that they result in faster care and/or better care (more clinically appropriate, safer and more evidence-based). The last paper in this series will address this question. But who are the real monetary beneficiaries of these systems. According to the next paper, it is estimated that 90% of this benefit accrues to the payors.

LEGAL, ETHICAL, AND FINANCIAL DILEMMAS IN ELECTRONIC HEALTH RECORD ADOPTION AND USE
Sittig, D.F., et al, Pediatrics 127(4):e1042, April 2011

Electronic health records (EHRs) enable the use of clinical decision support (CDS) systems and facilitate the management of patient data on both an individual and collective level, innovations necessary for successful health care reform. The authors, from the University
of Texas Health Sciences Center and Baylor College of Medicine, address legal, ethical, and financial questions that, if not answered, may prevent the widespread implementation and use of EHRs. Several legal dilemmas deserve prompt discussion, including the liability of providers who override CDS recommendations; the degree to which a clinician is responsible for examining a patient’s entire computerized record from multiple clinicians and institutions; and proprietary restrictions that prevent clinicians from publicly reporting safety concerns associated with EHRs. Ethical dilemmas bearing further discussion include the sale of de-identified patient information by EHR vendors, the right to opt out of electronic health information storage, and adolescents’ ability to control access to their data. Because a data-driven approach will inevitably be used to redesign the nation’s healthcare and payment systems, it is important to address who will have access to data from EHRs and who will oversee data aggregation, verification, analysis, and interpretation. The primary unresolved financial dilemma is who should pay for EHR adoption, because it is estimated that 90% of the monetary benefit from their use goes to health care payers rather than to those who use and own EHRs. The authors recommend that patients, clinicians, lawyers, ethicists, economists, policymakers, and informaticians promptly begin a national discussion of these issues. Solutions to these legal, ethical and financial dilemmas must be identified if EHRs are to be used to achieve higher-quality, lower-cost health care. 67 references (dean.f.sittig@uth.tmc.edu – no reprints)

Copyright 2011 by Emergency Medical Abstracts – All Rights Reserved 10/11 – #33
 
And what about quality? It just seems intuitive that HIT systems will increase the quality of patient care. Although the data in the following study is rather old, the authors from RAND and Harvard, in a study involving over 2,000 hospitals, failed to demonstrate that there was a clear relationship between use of HIT systems and improvement in quality measures for inpatient care.

ELECTRONIC HEALTH RECORD ADOPTION AND QUALITY IMPROVEMENT IN US HOSPITALS
Jones, S.S., et al, Am J Man Care 16(12):SP64, December 2010

BACKGROUND: The Health Information Technology for Economic and Clinical Health (HITECH) Act of 2009 offers substantial financial incentives to implement the use of electronic health records (EHRs), based on the unproved assumption that EHR use will lead to improved patient care.

METHODS: The authors, from RAND Corporation and Harvard, examined the association between EHR use in 2003 and 2006 and 17 quality-of-care indicators for acute myocardial infarction (AMI), congestive heart failure (CHF) and pneumonia in 2004 and 2007 in 2,021 US hospitals.

RESULTS: The percentage of participating hospitals with no EHR system decreased from 76% in 2003 to 62% in 2006. The number using a basic EHR system (consisting of a clinical data repository, electronic patient record and clinical decision support) increased from 22% to 26%, and the number using an advanced EHR system (the aforementioned components plus computerized physician order entry) increased from 2% to 12%. From 2004 to 2007, quality scores for all three conditions of interest improved, regardless of EHR use. Compared to hospitals with no EHR, hospitals with basic EHRs had greater improvement in quality measures for heart failure care (by 2.6%) but not for care of AMI or pneumonia, and quality scores in hospitals with advanced EHRs did not improve more or less than scores in hospitals without EHRs. Of note, quality scores for AMI and heart failure in hospitals that newly adopted an advanced EHR improved less than in hospitals that did not adopt EHR, and there was no relationship between adoption or upgrade of an EHR and quality improvement for pneumonia.

CONCLUSIONS: These data fail to show a clear relationship between use of EHRs and improvement in quality measures for inpatient care. 51 references (sjones1@rand.org – no reprints)

Copyright 2011 by Emergency Medical Abstracts – All Rights Reserved 7/11 – #10

So, it appears that it is not a given that HIT systems will allow patients to be treated more efficiently or better. While all other industries that have adopted computerization have clearly become more efficient and the quality of the work has become consistently better, medicine is not banking. Having the most expensive personnel in the building (doctors and nurses) pecking at computers to do data entry that could be done by others for a fraction of the cost just doesn’t make sense unless clinical decision support is refined to the point that it is truly helpful in medical decision making.

In settings where there are a high level of patient encounters per hour, the creation of medical records takes a disproportionate amount of time. The creation of medical records in these setting should be facilitated, not encumbered. Now physicians are creating “work arounds” to deal with their loss of productivity. Scribes are being hired to help neutralize the additional time required of HIT systems to generate patient charts – a substantial additional expense just to get back to the level of productivity that existed before the implementation of HIT systems.

But as noted above, despite a lack of compelling evidence regarding quality or efficiency, and despite their enormous costs, HIT systems that don’t communicate with each other and require increased provider time will continue to be rolled out on the assumption of value – value that has yet to be convincingly proven.