Scrutinizing the Dartmouth Atlas methodology

The worst-kept secret in journalism circles recently was that the New York Times was planning an article critical of the Dartmouth Atlas. That article was published online last night.

Among the main points in the article:

• “The mistaken belief that the Dartmouth research proves that cheaper care is better care is widespread.”

• “the atlas’s hospital rankings do not take into account care that prolongs or improves lives.”

• “Even Dartmouth’s claims about which hospitals and regions are cheapest may be suspect.”

• “failing to make basic data adjustments undermines the geographic variations the atlas purports to show.”

The Times has also published the correspondence it had with the Dartmouth team about methodology questions.

The Dartmouth team challenges each of these criticisms in a pdf statement online. The team says the Times made at least five factual errors and several misrepresentations. They write: “What is truly unfortunate is that the Times missed an opportunity to help educate the American public about what our research actually shows — or about the breadth of agreement about what our findings mean for health care reform.”

Oddly, the Times called the Atlas researchers “a once obscure research group at Dartmouth College.” I guess if you consider “once obscure” as 20 years ago you might be right. Putting that phrase in the opening sentence opened the discussion on a questionable note.

The Times is correct in stating that “The debate about the Dartmouth work is important.”

But as Merrill Goozner wrote on his blog today,

“…as is often the case in journalism, the attempt to reduce complex realities into a single-factor analysis that can be summarized in a headline or a single “why this story is important” paragraph can leave a mistaken impression. Regional variation in Medicare spending is one indicator of gross overutilization. Something is happening when a hospital in McAllen, Texas does twice as many knee implants per Medicare beneficiary as a hospital in Rochester, Minnesota.

But that by itself tells us nothing about why that overutilization occurs. Greedy doctors or hospitals working in a fee-for-service system are part of the problem. Cost shifting due to high levels of uninsurance and illness severity also may account for some or even much of those differences. And, as the Times report points out, quality differentials have to be taken into account.

The pushback by Dartmouth defenders has already begun. Columbia University professor Andrew Gelman on his blog suggests the reporters need a lesson in statistics. My complaint with today’s story comes from an entirely different angle. Higher quality care lowers costs, it doesn’t raise costs.”

Thoughtful people should read a lot about the criticism and the countering comments from Dartmouth. This is not something that can be absorbed in a headline, in a nut graf, or in a quickie talk show segment.

Disclosure: I worked at Dartmouth throughout the 90s, right down the hall from many of these Atlas researchers. I consider many of them my friends. My work is currently supported by the Foundation for Informed Medical Decision Making, one of whose co-founders was Dr. Jack Wennberg of Dartmouth, whose work was the basis for the Atlas project.

You might also like

Comments

Please note, comments are no longer published through this website. All previously made comments are still archived and available for viewing through select posts.

pnschmidt

June 3, 2010 at 11:02 am

The key to understanding the atlas comes from looking at the process for improving care implemented by another member of the Dartmouth team, Gerry O’Connor and his work in registries. In, for example, cystic fibrosis, the registry showed the discrepancies in outcomes. It did not show the reason for the discrepancies. (This is well covered in Atul Gawande’s book “Better” or his New Yorker article, “The Bell Curve.”) Knowing that discrepancies exist does not explain how to address them, but one needs this knowledge in order to generate the testable hypotheses that will address them. Dartmouth’s O’Connor, Wennberg, and others like Eugene Nelson have each led targeted projects to synthesize the outcomes data (like what’s in their atlas) with detailed analyses of the processes that led to those outcomes in order to improve care. Given this, the “atlas” is aptly named: a geographic atlas neither tells me where to go nor how to get there, it just informs my choices.

Rose Hoban, RN, MPH

June 3, 2010 at 9:24 pm

Another one of the implications of the article is that the Dartmouth data might be lacking because as the authors themselves ‘acknowledged in interviews that in fact it mainly shows the varying costs of care in the government’s Medicare program.’
Um. Maybe the Times reporters might have mentioned that there’s no other dataset out there where you can compare similar people across the country the way you can with Medicare beneficiaries. That’s because hospitals and insurers call their data ‘proprietary.’
Boy. Would I love to see Dartmouth Atlas methods applied across all age groups, income groups and all kinds of insurance statuses.