Scrutinizing the Dartmouth Atlas methodology

The worst-kept secret in journalism circles recently was that the New York Times was planning an article critical of the Dartmouth Atlas. That article was published online last night.

Among the main points in the article:

• “The mistaken belief that the Dartmouth research proves that cheaper care is better care is widespread.”

• “the atlas’s hospital rankings do not take into account care that prolongs or improves lives.”

• “Even Dartmouth’s claims about which hospitals and regions are cheapest may be suspect.”

• “failing to make basic data adjustments undermines the geographic variations the atlas purports to show.”

The Times has also published the correspondence it had with the Dartmouth team about methodology questions.

The Dartmouth team challenges each of these criticisms in a pdf statement online. The team says the Times made at least five factual errors and several misrepresentations. They write: “What is truly unfortunate is that the Times missed an opportunity to help educate the American public about what our research actually shows — or about the breadth of agreement about what our findings mean for health care reform.”

Oddly, the Times called the Atlas researchers “a once obscure research group at Dartmouth College.” I guess if you consider “once obscure” as 20 years ago you might be right. Putting that phrase in the opening sentence opened the discussion on a questionable note.

The Times is correct in stating that “The debate about the Dartmouth work is important.”

But as Merrill Goozner wrote on his blog today,

“…as is often the case in journalism, the attempt to reduce complex realities into a single-factor analysis that can be summarized in a headline or a single “why this story is important” paragraph can leave a mistaken impression. Regional variation in Medicare spending is one indicator of gross overutilization. Something is happening when a hospital in McAllen, Texas does twice as many knee implants per Medicare beneficiary as a hospital in Rochester, Minnesota.

But that by itself tells us nothing about why that overutilization occurs. Greedy doctors or hospitals working in a fee-for-service system are part of the problem. Cost shifting due to high levels of uninsurance and illness severity also may account for some or even much of those differences. And, as the Times report points out, quality differentials have to be taken into account.

The pushback by Dartmouth defenders has already begun. Columbia University professor Andrew Gelman on his blog suggests the reporters need a lesson in statistics. My complaint with today’s story comes from an entirely different angle. Higher quality care lowers costs, it doesn’t raise costs.”

Thoughtful people should read a lot about the criticism and the countering comments from Dartmouth. This is not something that can be absorbed in a headline, in a nut graf, or in a quickie talk show segment.

Disclosure: I worked at Dartmouth throughout the 90s, right down the hall from many of these Atlas researchers. I consider many of them my friends. My work is currently supported by the Foundation for Informed Medical Decision Making, one of whose co-founders was Dr. Jack Wennberg of Dartmouth, whose work was the basis for the Atlas project.