That Dogma Won’t Hunt

January 4, 2010 at 9:01 pm 25 comments

“Science is properly more scrupulous than dogma.  Dogma gives a charter to mistake, but the very breath of science is a contest with mistake, and must keep the conscience alive.”

- George Eliot, Middlemarch

In the past year there’s been a lot of buzz about a University of Pennsylvania study that’s been quoted extensively as proving that “confrontational training methods” provoke aggressive responses in dogs. All the cool kids have been citing it, but being a skeptic I couldn’t help but wonder — does this study really prove what they say it does?

In the introduction to the study Meghan Herron, Frances Shofer, and Ilana Reisner (Herron et al.) write that:

The purpose of this study was to describe the frequency of use, the recommending source, and the owner-reported effect on canine behavior of interventions that owners of dogs with undesired behaviors had used on their dogs. This study also aimed to report aggressive responses from the dogs subsequent to the use of aversive and non-aversive interventions.

If you jump to the end of the study where the investigators discuss their conclusions you’ll see that Herron et al. do not expressly claim to have proved anything. They merely state that an association between the use of “confrontational training methods” and aggressive canine responses was observed in the population sampled. Confusion about that silly cause-effect thingamahickey might have occurred because some PR hack inflated the results to sex-up a press release. Or… it may have come about because immediately after Herron et al. note that they observed an association, they go on to state that: “Ultimately, reward-based training is less stressful or painful for the dog, and, hence, safer for the owner.”

[scratches head] I believe that this is akin to saying: “We found that being black in color was associated with a higher risk of euthanasia in the shelter dog environment. Ultimately, being white in color is less stressful or dangerous for a dog, and, hence, breeders should strive to produce only white dogs.”

How does a simple association suddenly morph into a cause that requires intervention? Correlational studies indicate the existence of relationships between  variables, but they can’t prove that one variable causes a change in another.  Correlated variables may not be linked in time, space or function, and factors that weren’t considered in the study may be important – or even causative. I come from a different world so maybe I’m being picky, but in the physical sciences it is considered quite bad form to use a simple correlation to imply one has discovered a cause-effect relation.

Unfortunately – the biggest problem in science today isn’t that it’s hard to understand – it’s how distressing easy it has become to use it to advance a personal, economic or political agenda. In some circles, using science to advance an agenda has become so common that it seems that researchers are engaging in this kind of misuse of science without even being aware of it. As Dr. Free-Ride wrote over at Adventures in Ethics and Science:

“If they are completely committed to a particular view of the world, fully expect that the evidence will support that view, and blame mistaken methodology for results that don’t support that view — indeed, to the point of openly rejecting evidence-based medicine and scientific standards of proof — the problem isn’t one of lying so much as arguing for a different standard of credibility.”

Good science and dogma don’t mix.

And what about the science? Well, to begin with, there are several bias problems in the survey population. The first one I found, and in my opinion it’s a rather glaring one, is that the study population only included owners who were actively seeking help from a veterinary behavior clinic. Given this limitation, all dog owners who found that aversive methods worked effectively for them would be removed from the study population.

Herron et al. don’t mention anywhere the strong and, I think obvious, likelihood that the population selected was also likely to include a very high percentage of people who were frustrated with their dogs. Frustrated owners are a lot more likely to lash out at their dogs in anger — something no trainer I’ve met would consider a valid “training method”. Including this kind of angry, frustrated behavior in the same category as a thoughtful, well-timed leash correction makes about as much sense as calling free-feeding a “reward-based training method”.

Herron et al. also don’t discuss how many of the dogs included in the survey exhibited aggressive behavior before implementation of the training methods. If aggressive behavior preceded implementation of a training method, that training method should not be considered the cause of the aggressive behavior because, by definition, cause must occur before effect.

Sampling a non-representative population may not be good science, but it can be a nifty way to screen results to justify preconceived notions.

The training methods included in the study are described only vaguely and Herron et al. do not state why each method is considered to be confrontational, neutral or nonconfrontational. For some reason “leash correction” and use of a “choke or pronged pinch collar” are considered separately, as are “dominance down” and “force down”. Three different kinds of verbal corrections are evaluated separately, and two different trained commands (“sit to get everything” and the “watch” attention cue) appear to have been included in the “non-aversive” group without considering what kinds of methods were used to train the dogs to obey these commands.

The discussion section does not present a balanced presentation of the training methods studied.  Detailed discussions on the potential negative aspects of some of the aversive techniques were included, but for some reason no information was presented on any potential positive effects of these techniques. No discussion of potential negative or positive effects of the nonconfrontational or neutral methods was included.

The most obvious examples of this is the detailed (and IMO rather convoluted) justification presented to discredit all use of shock collars.  A similar discussion is provided regarding of the use collar corrections. I’m not sure why Herron et al. felt obliged to include detailed discussions on the possible adverse effects of these methods and no others, though I think it is important to note that leash corrections were ranked by owners as the fifth most effective method (of the thirty studied) and resulted in aggressive responses in roughly the same percent of cases as the positive / neutral methods “using food to trade for item” and “avoidance”. The use of “shock collars” resulted in the same low percentage of aggressive responses (4-6%).

Intended or not, the unbalanced discussion of methods casts a strong appearance of bias on this study.

I found the study to be somewhat confusing because in places the discussion does not appear to match the results obtained.  While Herron et al. describe the use of Cesar Millan’s characteristic ‘schhhtt’ sound and “jabbing at the dog in the neck” as “potentially provocative” – according to their own data these methods elicited aggressive responses in the same percentage of cases as the use of food rewards.  Given these results, I do not understand why Millan’s methods are considered to be provocative and the use of food rewards is not.

Herron et al. state that owners of dogs with aggression problems who consult dog trainers instead of veterinarians are at risk because “the lack of standardized oversight of many training programs has resulted in a range of competence and ethical practice of behavior modification and owners may be at risk of receiving unsafe advice”. I find it interesting, given this statement, that so many trainers have latched onto this study as gospel truth.

In the sentence immediately after the one where Herron et al. recommend that veterinarians be consulted instead of trainers, they state that the most common intervention recommended by veterinarians is the use of a muzzle. In an interesting bit of irony — this was the method that resulted in the highest percentage of aggressive responses.

In the conclusions Herron et al. note that “confrontational or aversive behavioral interventions applied by dog owners before their pets were presented for a behavior consultation were associated with aggressive responses in many cases” and that “reward-based training is less stressful or painful for the dog”. I am not at all convinced that the study demostrated this, and I can’t help but wonder if a bias to support positive methods and condemn aversive ones polluted this study.

I find it disturbing that a study with so many obvious holes in it is being touted as “accepted science” throughout the media. Herron et al’s “Survey of the use and outcome of confrontational and non-confrontational training methods in client-owned dogs showing undesired behaviors“:

  • Sampled a heavily biased population;
  • Includes a discussion section that is heavily slanted against all use of aversive/confrontational methods even though this opinion does not appear to have been supported by the survey data;
  • States a conclusion that is not supported by the data; and
  • Presents correlation as cause, and then uses the inferred cause to manufacture a problem and a sense of urgency to address it.

I’m afraid that instead of presenting a balanced, scientifically rigorous evaluation of the data they collected, Herron et al. presented the results they wanted — and expected — to find. Whether you agree with their philosophy or not, this was not good science.  Good science involves a commitment to follow the data, even when that data leads you in a direction you’re not ideologically comfortable with.

It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.

Sherlock Holmes, “A Scandal in Bohemia”

Entry filed under: behavior science, dog training, dogs. Tags: .

Complete Sentences Conflicting Conclusions

25 Comments Add your own

  • 1. Viatecio  |  January 5, 2010 at 1:41 am

    Excellent review!

    I think the thing I find the most infuriating is that so-called “provocative” methods are put in such all-inclusive lists that some things true balanced trainers recommend are lumped in with stuff that NO trainer (or at least one interested in the well-being of the dog) would recommend today, and even combined with stuff that IS recommended (such as the ever-popular “squirt bottle” ‘punishment’ so frequently dished out by ‘hands-off’ trainers). These people must have their heads stuck up the Koehler era’s arse, because there is simply no explanation for such a hodgepodge list, that is, if you count out the truely unprofessional trainers who harm more dogs than they actually help.

    It’s sad how good science is twisted by political agendas and warm fuzzies.

    One of the new dogs in the school kennel is very strong and dog-aggressive (oh yeah, she’s a “pit-mix”), to the point where she will fence-fight other dogs through their doors when she is being taken out. But she’ll do just fine, because (as the attending vet/’behaviorist’ stated) “She’s been fitted with a Gentle Leader and will be taken outside through another door so she won’t have access to any of the other dogs.” Must…resist…sneaking in…my Dogtra…

    Oh, and according to said vet/’behaviorist,’ the only people who can help with aggression and behavior problems are “BEHAVIORISTS” who have a PhD in animal behavior or who are DVMs. Because, you know, “Trainers” just take care of that “sit-stay stuff.” PFFFFFFTHBTHBTHB is what I say, to myself of course, because I’m just the lowly student and not able to say anything like “Well, wait a second here…”

  • 2. H. Houlahan  |  January 5, 2010 at 2:44 am

    Well, I’ve been sitting on my own dissection of this drivel for a year, so I can’t exactly complain that you scooped me.

    Carry on.

  • 3. Christopher@BorderWars  |  January 5, 2010 at 3:29 am

    I’m of the efficacy school of dog training, i.e. whatever works best. I don’t buy into some dogma about talking too harshly to fluffy being offensive and all that.

    Seeing all the points you raise, I’ve thought of a few more. For instance, if the entire list of participants are linked with a single behaviorist, we have horrible sample bias for multiple reasons:

    1) People who seek the help of a behaviorist, as you said. Aren’t you already buying into a lot of dogma by going to see a behaviorist in the first place? It’s rather like sampling only people who go to chiropractors or acupuncturists… there’s already a rather significant selection bias present in the clients who walk through the door.

    2) People who chose THIS ONE behaviorist. What was the advertising like? Does this one behaviorist have a particular philosophy/agenda and attract like-minded clients? Were these all new clients or continuing clients? What information have they already been given by their behaviorist?

    3) Self reporting. Any science that relies on surveys instead of measured data is suspect. You must trust that all of your participants are equally informed, clear on what the questions are asking and have the ability to judge what actually happened with clarity. NOT LIKELY!

    I recall a middle school science lab where the professor had a dozen or so items around the room that we had to measure individually, things like what color is this litmus strip, etc. The more subjective the judgment, the less agreement. And this was observing the same exact event.

    4) People-pleaser effect. Most people appreciate that in drug trials people who take sugar pills will mind-over-matter their way to better feelings (placebo effect). But there’s also the element that test subjects are very smart and aim to please when participating in an official study.

    If the questions were written with suggestive language, it’s pretty easy to ascertain the researcher’s bias and aim to please.

    From what you’ve stated, it doesn’t sound like this study was even able to make any conclusions from its inception and is rather a big waste of time and money.

    Without looking at the results, could you answer this question: Does the procedure add clarity to the question at hand? In this case, I’d say no.

  • 4. YesBiscuit!  |  January 5, 2010 at 12:25 pm

    You left out THE most important point by those touting the study as being “proof” that Millan’s methods are bad: We hatezez teh Seezr Meelawn!111!!!!!!!!11

  • 5. Catherine Nelson  |  January 5, 2010 at 4:31 pm

    A fifth grader could determine that this study doesn’t qualify as science at all — so how uninformed IS the general public about what constitutes science, that every journalist seems merely to repeat this “finding”? Do they assign only the stupid ones to the dog stories? And what does it say about the freethinking abilities of those who simply accept and repeat this as Truth? (including, by the way, Dr. Nicholas Dodman)

    But more sadly, what about young Ms. Herron, getting a worthless Ph.D.? Of course no respectable funding institution would have anything to do with this dreck. The only future for such a person is to perpetuate tired political opinions among the like-minded.

    Fortunately (brightening up), it’s such *obvious* crap that no one in the real biological sciences (including those who study behavior), will take it seriously. Science always wins.

  • 6. Jill  |  January 5, 2010 at 4:48 pm

    A delicious parsing of the argument(s). So much so that I’m thinking of using both (the original ‘research’ article alongside this close reading) in my College Writing II course this term.

  • 7. Pooch Professor  |  January 6, 2010 at 3:31 am

    Beautiful dissection of the study. I read it after a day wherein I attempted to explain on a professional trainers list why I do not simply accept as fact most of the conclusions about “positive training.” Namely, that no credible scentific study exists that proves the proper use of punishment or “aversives” is harmful to dogs. Not one.

    As for dogma, I have long believed that clinging to the notion that “positive training” is the only humane way to instruct dogs is not much different than clinging to overzealous religious fundamentalism. And we all know tht no one is more zealously annoying than the converted.

  • 8. Eowyn  |  January 6, 2010 at 8:15 pm

    Really nice discussion. As both a dog trainer and a professional economist who has spent much of my life analyzing studies, it is SO frustrating to see bad science touted as definitive of something. Keep up the good work!

  • 9. Melissa Jo Peltier  |  January 7, 2010 at 5:45 am

    Excellent analysis of what is not an actual “study” conducted under the guidelines of the Scientific Method, but instead, a “survey,” designed to produce the same biased and variable results that it describes. Unfortunately, it uses the title of a prestigious University to give it undeserved weight.
    All your points are right on the money. In addition, as another commenter so wisely pointed out, all the owner’s results are “self-reported.” Nor is there any control group or sample, or any other way by which another scientist could attempt to replicate the results of the first “study.”
    Thanks for this very well-thought out argument. (And great title, too!)

  • 10. SmartDogs  |  January 7, 2010 at 6:04 am

    Congratulations on your team’s People’s Choice Award!

  • [...] about a study of the subject, here's another view of that study from someone who read the study… That Dogma Won’t Hunt Smartdogs' Weblog I would read the study myself, but I'm not going to pay a subscription to read it, so I cannot [...]

  • 12. Mongo  |  February 15, 2010 at 12:33 am

    Thank you for your thoughtful dissection of the study.
    I actually read the whole stupid thing months ago.
    I still could not figure out why such vague crap was being offered as “proof” of anything.

    There was another one chasing that “study” that “proved” many dogs being euthanized for aggression had been”trained” by owners using hitting and kicking (and it was refered to as “training”!!!!) It was the perfect follow up article by P+ posters who felt any aversive physical contact was the same as a hard kick and slap. sigh

    Personally, I think it was latched onto by many P+ ONLY training folks and blasted to every online message board, e group and list because
    A) P+ONLY trainers feel the need to validate their ideas to those more rational and experienced
    B) P+ ONLY trainers are often too naive to question a “scientic study”.
    So the story had legs it didn’t deserve.

    Being read online doesn’t make something more valid any more than being on YouTube makes something actually news worthy.
    just my .02

  • 13. SmartDogs  |  February 15, 2010 at 10:17 am

    Thanks, I’ll have to look for the chaser. It may be worth a poke or two as well.

  • 14. ruthcrisler  |  March 21, 2010 at 7:38 am

    I read this study myself when it came out, and was very disappointed to see what passes for “science” in the press today.

    Thank you for setting down such a thorough, crisp, and entertaining analysis. I am grateful, and also a little jealous (your title for the piece is probably the best name for anything I’ve ever encountered–wish I’d thought of it first).

    BTW, did you notice that this survey of what, a hundred dog owners, was actually billed as a “year-long study” when it was released?!

  • 15. SmartDogs  |  March 21, 2010 at 6:58 pm

    I’ve noticed that most of the really crappy “passes for science” in the world of dogs comes out of the U.S. In Europe, especially eastern Europe, they are doing some really interesting things. I don’t know if this is a relict of the American behaviorism vs European ethology rift or if the screaming mouths of popular dogmatic trainers here have succeeded in silencing most dissenting research on this side of the pond.

    Or something else altogether.

  • [...] undesired behaviors, Applied Animal Behavior Science 117: 47-54. I reviewed this article in detail in a previous post. Even though Herron et al. present a somewhat detailed literature survey of dog training methods [...]

  • 17. Dega Vans  |  April 17, 2011 at 8:56 pm

    I read this in IACP’s Safehands Journal today. My trainer and I got a real kick out of it. You see my last chance rescue dog was saved by tradional prong collar and praise training. It started with a dog who was lungeing at people and small dogs. This dog went for a year with all the “no adversives” training in the book and was only getting worse. Worse to the point of living in a muzzle and being confined to the house.

    As a “last last chance” I changed over to tradional training, where he is corrected for misbehavior, and a year later I have a whole new dog. He is building his off-lead reliablity, is accepting friendly stangers, is non-reactive and even friendly to other dogs, and is a real joy to take out in the world with me. Balenced, happy, stable dog with a forever home and no chance of being put down for behavior issues.

    Of course I was not in their study group on the affects of “adversives” in training.

  • 18. Another interesting blog  |  June 13, 2011 at 5:08 pm

    [...] interesting blog That Dogma Won’t Hunt Smartdogs' Weblog I'm really curious after reading this blog and the comments below… Regardless of whether or not [...]

  • 19. Some Links for Your Enjoyment - DogProblems.com  |  July 27, 2011 at 4:45 pm

    [...] That Dogma Won’t Hunt : a critical examination of one of the key studies many behaviorists point to when they insist that punishment is counterproductive in dog training [...]

  • [...] That Dogma Won’t Hunt : a critical examination of one of the key studies many behaviorists point to when they insist that punishment is counterproductive in dog training [...]

  • 21. Gary Wilkes  |  October 13, 2011 at 2:01 am

    Nice work, Janeen. I guess we should all remember that behavioral science isn’t really science – it is Kabuki theater that allows a few people to leverage the discussion. Learning theory doesn’t stand up the scientific definition of theory because the bulk of the hypotheses have never been proven and in most cases contradict objective observation of nature – a huge problem. Meaning, it’s not really supposed to be scientific, it must simply be scientistic…a word coined by critics of Skinner’s behaviorism meaning “having the appearance of science without the objective analysis demanded by real science.” So, along we go. The AVSAB position statement on punishment is also fun to parse. They cite Nate Azrin three times to prove their anti-punishment position. His very early work. His conclusion was that best way to get rid of unacceptable behavior is wiht some negative consequences – far faster than teaching an alternate behavior or developing stimulus control. They dont quote Azrin’s later work for some strange reason…Cherry picking is a very big part ot his …

  • 22. That Dogma Won’t Hunt « dogshepherd  |  May 9, 2012 at 4:57 pm

    [...] Janeen McMurtrie from http://smartdogs.wordpress.com/2010/01/04/that-dogma-wont-hunt/ - and The IACP SafeHands Journal ) “Science is properly more scrupulous than dogma.  Dogma [...]

  • 23. That Dogma Won’t Hunt « dogshepherd  |  May 9, 2012 at 4:57 pm

    [...] Janeen McMurtrie from http://smartdogs.wordpress.com/2010/01/04/that-dogma-wont-hunt/ - and The IACP SafeHands Journal ) “Science is properly more scrupulous than dogma.  Dogma [...]

  • 24. metisrebel  |  September 4, 2012 at 8:45 am

    I’ve been arguing against this “junk science” for longer than I care to admit.

    Thank you for wading through all these studies so I can just add links and tell people “THAT’S WHY!” ;)

  • 25. Deferred Suffering | diaryofadogpusher  |  September 9, 2013 at 4:31 pm

    […] And hat tips to master trainers who have tackled this subject before with much more skill, eloquence, and tenacity than I’m capable of: http://www.seespotrunkennel.com/blog/what-i-mean-by-balanced/; http://www.seespotrunkennel.com/blog/undue-temperance/; http://smartdogs.wordpress.com/2010/01/04/that-dogma-wont-hunt/ […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Trackback this post  |  Subscribe to the comments via RSS Feed


Because A Dog’s Mind is a Terrible Thing to Waste

RSS

Copyright notice

All original content on this Web site is copyright © on the date of publication by this author. All rights reserved except, of course, that others may quote from original content under the 'Fair Use' provisions of US copyright law.

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 272 other followers

Top Dog Blog
Featured in Alltop
ResearchBlogging.org

sitemeter

Add to Technorati Favorites
Dog Blogs - BlogCatalog Blog Directory
blogarama - the blog directory
Blog Directory
Blog Directory & Search engine
January 2010
M T W T F S S
« Dec   Feb »
 123
45678910
11121314151617
18192021222324
25262728293031

Follow

Get every new post delivered to your Inbox.

Join 272 other followers

%d bloggers like this: