Saturday, May 03, 2008

A Nice, Clear Critique of Hypothesis Testing

Students at times come to my office to tell me that their professors in research methods courses emphasize the importance of null hypotheses and hypothesis testing. It always brings to mind the last words Gary Chamberlain said in his econometrics class: "I didn't mention hypothesis testing, because I don't like it. And don't ever use those damned stars." [I am, of course, paraphrasing, as I took the class 25 years ago].

In any event, I ran across this nice critique of hypothesis testing. The point of statistics, after all, should be parameter estimation and prediction, not some arbitrary test of statistical significance. In large samples, statistically significant parameters can be unimportant; in small samples, statistically insignificant parameters can simply mean that we don't really know what is going on.

Much of the confusion underlying how medical information is conveyed in the media arises from the hypothesis testing tradition: the media infer that if x is a statistically significant predictor of y, then x must be important in determining y. This is not necessarily so.


Austin Kelly said...

I've always found academics strangely biased towards hypotheses testing as opposed to decision theory, even though I was always taught by my 'metrics profs to distinguish between 'importance' and 'statistical significance.'

My two favorite personal experiences are from work I did on capitalization of above market financing and down payment assistance. I noticed that the literature on capitalized financing invariably tested the null hypothesis of 'beta=0' even though theory implied 'beta=1.' I beleive everyone wanted to make the easy case 'hey we got a significant coefficient' rather than address the issue of 'hey we got a different coefficient than theory implies.' In the down payment assistance case GAO found that in a small sample there was about an 85% probability that these things cost FHA money, and Stephen Fuller responded to the study by indicating that we haven't proven that it was the case at 95% probability. I looked at the critique and thought to myself "there's an 85% chance that they are losing money and they shouldn't do anything because it isn't 95%???? - what does decision theory say about that?" Anyway, later with an expanded dataset I did find a greater than 95% probability, so I guess that can finally be put to bed.

Phyliss said...

Interesting to know.

Anonymous said...



polo shirts said...

History of polo ralph lauren. Polo fashions had its humble beginnings in 1968 when tie salesman Ralph Lauren gave it a kick start. By 1969 he had a boutique polo ralph lauren factory stores within the Manhattan department store Bloomingdale's. ... Brands and luxury standard. Since Ralph Lauren's first brand, Polo Ralph Lauren, was launched, the company has expanded to include a variety of luxury brands such as Polo Golf, Polo Denim, Polo Sport. You can buy cheap Ralph Lauren Clothing at Ralph Lauren outlet.Also We provide polo shirts
Ralph Lauren polo shirt, 50% OFF! polo ralph lauren outlet online is your best choice!In 2006, polo ralph lauren outlet became the first designer in Wimbledon's 133-year history to create official uniforms for the tournament. As part of this year's event, which starts next week, polo ralph lauren sale will introduces the first ... determination to maintain and enhance the values for which our two brands are famous throughout the world. The rugby ralph lauren brand brings to Wimbledon the look of timeless elegance, drawing on our rich history and traditions

gaohui said...

You will not have ed hardy clothes to worry about any Christian audigier discomfort, and best of all; most are ed hardy shoes designed to fit your body as the ed hardy outlet little bundle of joy grows allowing ed hardy Bikini you to wear the maternity clothing more than ed hardy hats once.The development of holiday maternity ed hardy swimsuits clothing has coincided with the changing attitudes of ed hardy clothing women regarding their bodies. Today's ed hardy glasses holiday maternity fashions are ed hardy trendy, stylish and sexy, fitting today's ed hardy iphone cases fast-paced lifestyle Now, women ed hardy dresses are proud to show off their baby bump while wearing ed hardy Jackets striking and classy maternity clothing.