That Dogma Won’t Hunt

“Science is properly more scrupulous than dogma.  Dogma gives a charter to mistake, but the very breath of science is a contest with mistake, and must keep the conscience alive.”

– George Eliot, Middlemarch

In the past year there’s been a lot of buzz about a University of Pennsylvania study that’s been quoted extensively as proving that “confrontational training methods” provoke aggressive responses in dogs. All the cool kids have been citing it, but being a skeptic I couldn’t help but wonder — does this study really prove what they say it does?

In the introduction to the study Meghan Herron, Frances Shofer, and Ilana Reisner (Herron et al.) write that:

The purpose of this study was to describe the frequency of use, the recommending source, and the owner-reported effect on canine behavior of interventions that owners of dogs with undesired behaviors had used on their dogs. This study also aimed to report aggressive responses from the dogs subsequent to the use of aversive and non-aversive interventions.

If you jump to the end of the study where the investigators discuss their conclusions you’ll see that Herron et al. do not expressly claim to have proved anything. They merely state that an association between the use of “confrontational training methods” and aggressive canine responses was observed in the population sampled. Confusion about that silly cause-effect thingamahickey might have occurred because some PR hack inflated the results to sex-up a press release. Or… it may have come about because immediately after Herron et al. note that they observed an association, they go on to state that: “Ultimately, reward-based training is less stressful or painful for the dog, and, hence, safer for the owner.”

[scratches head] I believe that this is akin to saying: “We found that being black in color was associated with a higher risk of euthanasia in the shelter dog environment. Ultimately, being white in color is less stressful or dangerous for a dog, and, hence, breeders should strive to produce only white dogs.”

How does a simple association suddenly morph into a cause that requires intervention? Correlational studies indicate the existence of relationships between  variables, but they can’t prove that one variable causes a change in another.  Correlated variables may not be linked in time, space or function, and factors that weren’t considered in the study may be important – or even causative. I come from a different world so maybe I’m being picky, but in the physical sciences it is considered quite bad form to use a simple correlation to imply one has discovered a cause-effect relation.

Unfortunately – the biggest problem in science today isn’t that it’s hard to understand – it’s how distressing easy it has become to use it to advance a personal, economic or political agenda. In some circles, using science to advance an agenda has become so common that it seems that researchers are engaging in this kind of misuse of science without even being aware of it. As Dr. Free-Ride wrote over at Adventures in Ethics and Science:

“If they are completely committed to a particular view of the world, fully expect that the evidence will support that view, and blame mistaken methodology for results that don’t support that view — indeed, to the point of openly rejecting evidence-based medicine and scientific standards of proof — the problem isn’t one of lying so much as arguing for a different standard of credibility.”

Good science and dogma don’t mix.

And what about the science? Well, to begin with, there are several bias problems in the survey population. The first one I found, and in my opinion it’s a rather glaring one, is that the study population only included owners who were actively seeking help from a veterinary behavior clinic. Given this limitation, all dog owners who found that aversive methods worked effectively for them would be removed from the study population.

Herron et al. don’t mention anywhere the strong and, I think obvious, likelihood that the population selected was also likely to include a very high percentage of people who were frustrated with their dogs. Frustrated owners are a lot more likely to lash out at their dogs in anger — something no trainer I’ve met would consider a valid “training method”. Including this kind of angry, frustrated behavior in the same category as a thoughtful, well-timed leash correction makes about as much sense as calling free-feeding a “reward-based training method”.

Herron et al. also don’t discuss how many of the dogs included in the survey exhibited aggressive behavior before implementation of the training methods. If aggressive behavior preceded implementation of a training method, that training method should not be considered the cause of the aggressive behavior because, by definition, cause must occur before effect.

Sampling a non-representative population may not be good science, but it can be a nifty way to screen results to justify preconceived notions.

The training methods included in the study are described only vaguely and Herron et al. do not state why each method is considered to be confrontational, neutral or nonconfrontational. For some reason “leash correction” and use of a “choke or pronged pinch collar” are considered separately, as are “dominance down” and “force down”. Three different kinds of verbal corrections are evaluated separately, and two different trained commands (“sit to get everything” and the “watch” attention cue) appear to have been included in the “non-aversive” group without considering what kinds of methods were used to train the dogs to obey these commands.

The discussion section does not present a balanced presentation of the training methods studied.  Detailed discussions on the potential negative aspects of some of the aversive techniques were included, but for some reason no information was presented on any potential positive effects of these techniques. No discussion of potential negative or positive effects of the nonconfrontational or neutral methods was included.

The most obvious examples of this is the detailed (and IMO rather convoluted) justification presented to discredit all use of shock collars.  A similar discussion is provided regarding of the use collar corrections. I’m not sure why Herron et al. felt obliged to include detailed discussions on the possible adverse effects of these methods and no others, though I think it is important to note that leash corrections were ranked by owners as the fifth most effective method (of the thirty studied) and resulted in aggressive responses in roughly the same percent of cases as the positive / neutral methods “using food to trade for item” and “avoidance”. The use of “shock collars” resulted in the same low percentage of aggressive responses (4-6%).

Intended or not, the unbalanced discussion of methods casts a strong appearance of bias on this study.

I found the study to be somewhat confusing because in places the discussion does not appear to match the results obtained.  While Herron et al. describe the use of Cesar Millan’s characteristic ‘schhhtt’ sound and “jabbing at the dog in the neck” as “potentially provocative” – according to their own data these methods elicited aggressive responses in the same percentage of cases as the use of food rewards.  Given these results, I do not understand why Millan’s methods are considered to be provocative and the use of food rewards is not.

Herron et al. state that owners of dogs with aggression problems who consult dog trainers instead of veterinarians are at risk because “the lack of standardized oversight of many training programs has resulted in a range of competence and ethical practice of behavior modification and owners may be at risk of receiving unsafe advice”. I find it interesting, given this statement, that so many trainers have latched onto this study as gospel truth.

In the sentence immediately after the one where Herron et al. recommend that veterinarians be consulted instead of trainers, they state that the most common intervention recommended by veterinarians is the use of a muzzle. In an interesting bit of irony — this was the method that resulted in the highest percentage of aggressive responses.

In the conclusions Herron et al. note that “confrontational or aversive behavioral interventions applied by dog owners before their pets were presented for a behavior consultation were associated with aggressive responses in many cases” and that “reward-based training is less stressful or painful for the dog”. I am not at all convinced that the study demostrated this, and I can’t help but wonder if a bias to support positive methods and condemn aversive ones polluted this study.

I find it disturbing that a study with so many obvious holes in it is being touted as “accepted science” throughout the media. Herron et al’s “Survey of the use and outcome of confrontational and non-confrontational training methods in client-owned dogs showing undesired behaviors“:

• Sampled a heavily biased population;
• Includes a discussion section that is heavily slanted against all use of aversive/confrontational methods even though this opinion does not appear to have been supported by the survey data;
• States a conclusion that is not supported by the data; and
• Presents correlation as cause, and then uses the inferred cause to manufacture a problem and a sense of urgency to address it.

I’m afraid that instead of presenting a balanced, scientifically rigorous evaluation of the data they collected, Herron et al. presented the results they wanted — and expected — to find. Whether you agree with their philosophy or not, this was not good science.  Good science involves a commitment to follow the data, even when that data leads you in a direction you’re not ideologically comfortable with.

It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.

Sherlock Holmes, “A Scandal in Bohemia”