Much of the research that we read in medical journals should not be taken at face value, because of a series of errors in study design, analysis (often involving misuse of statistics), and/ or interpretation or extrapolation. By understanding a few basic concepts we can become adept at interpreting the validity of most of what we read.
Understanding Literature Bias part 1 in a series
Much of the research that we read in medical journals should not be taken at face value, because of a series of errors in study design, analysis (often involving misuse of statistics), and/ or interpretation or extrapolation. By understanding a few basic concepts we can become adept at interpreting the validity of most of what we read.
Bias is the systematic introduction of error, which distorts the results of a study in a non-random way. When the bias concerns the particular study population itself, that threatens the internal validity of the study. When the bias is related to the way in which the study data is applied to the larger population, that threatens the external validity of the study.
For this issue, let’s discuss “Internal Validity” and specifically, how referral bias creates issues with internal validity.
Internal Validity: Referral Bias
Ways in which internal validity is threatened can be broken down into several subsets. One of the most typical of these is referral bias (also sometimes called recruitment bias or selection bias). Suppose an investigator offered free psychiatric counseling to volunteers willing to participate in a study of adult women who had had an abortion, and then presented data to show that on MMPI testing they had a higher incidence of depression than the general population. The conclusion that there is a relation between depression and previous abortion would be invalid, because the nature of the recruitment (using psychiatric counseling as an inducement) would obviously be most likely to attract, from the entire group of women who had once had an abortion, those who were depressed. They would not be a representative sample of all women with a previous abortion.
A less obvious example, with great relevance to emergency medicine, comes from the early, landmark UPET study of pulmonary embolism. In this study patients sent to pulmonary angiography because of clinical suspicion of PE were evaluated vis-a-vis clinical findings, and those who proved to have a PE were found to differ from those who did not in several ways. Surprisingly, these included a lesser frequency of clinical signs of DVT. The authors concluded that the hypothesized association between DVT and PE is erroneous, and that DVT actually weighs against a diagnosis of PE.
The problem with this conclusion has mostly to do with referral bias. This is not a study of all people in the universe, but rather of those with clinical suspicion of PE. Since we know from pathology data that DVT is associated with, and is a cause of PE, the clinicians in this study undoubtedly were more likely to refer patients to angiography if they had “evidence” of DVT than if they didn’t. Thus, it is likely that many patients were referred merely because of such “evidence” (although in fact they were probably misclassified, since DVT is a notoriously unreliable clinical diagnosis), despite absence of other typical clinical findings or risk factors, and in fact had neither true DVT nor PE. This would inevitably lead to overrepresentation of “evidence of DVT” in the negative angiography group. On the other hand, since typical teaching has it that patients don’t get PE without DVT, the clinicians in the study may well have only referred patients to angiography, in the absence of “DVT”, if the rest of the picture was extremely suggestive of PE (i.e. history of cancer or recent pelvic surgery, tachypnea and dyspnea, etc), leading to a high likelihood of positive angiograms in patients with “absence of DVT.”
Referral bias can take other forms as well. Numerous textbooks claim that cat bites are more likely to get infected than dog bites. This is based on the consistent finding, in studies of bite wounds, that infection occurs in a higher percentage of those presenting with cat bites. But rather than assume that this is because of a truly higher predilection for infection in such cases, we might hypothesize that not all patients with bite wounds present to doctors and that the reasons for self-referral, which are likely to be different between victims of dog vs cat bites, may greatly impact these findings.
Cat-bites are often puncture wounds, which produce little gross structural damage, while many dog-bites produce significant lacerations that clearly require suturing. Perhaps the vast majority of cat bites are ignored, and patients come to an ED in most cases only if the wound becomes infected. If that were the case, it might be that despite an exceedingly low infection rate for all cat bites we would experience (and tabulate) a much higher infection rate in those patients we see in the ED. On the other hand, since many dog-bite wounds need suturing, victims often don’t wait for signs of infection before presenting, and even if there were a relatively higher incidence of infection following dog bites (which there probably isn’t), we would see what seemed to be a lower total rate among those presenting for care.
Finally, another example of referral bias is the influential case series of the British neurosurgeon Duffy, who reported multiple cases of brain herniation following LPs. What a group from Montreal discovered upon reviewing these cases was that most of them were deathly ill, with advanced brain tumors and most if not all were actually showing signs of herniation even before the LP. They were all Duffy’s patients. Since in those days neurosurgeons were even fewer and further between than today, virtually the only patients someone like
Duffy saw were those with end-stage neurosurgical disease. Extrapolating from such patients to the overwhelming majority of patients who get LPs for other reasons is of course absurd, but this did not prevent most of us from believing that an LP is a dangerous invasive test on the basis of virtually no other evidence than this one case series unless there is a prior CT, despite overwhelming evidence to the contrary.
Emergency physicians see the effects of this type of referral bias every day, whenever they speak to consultants. Since neurosurgeons only see the occasional head trauma patient with something more than the typical minimal signs and symptoms, they have a very skewed notion of what head trauma is, and suspect every bump on the head deserves a CT, because they see a very selected sub-sample in whom the incidence of significant injury is relatively very high.
Jerome Hoffman, MD Professor at the UCLA School of Medicine; Faculty at the UCLA ED; Associate Medical Editor for Emergency Medical Abstracts