Uncovering a Four Billion Dollar Statistical Mistake

In 1999, the State of New Mexico filed suit against the DOJ, General Electric, the US Air Force, and others. It asked for more than four billion dollars to compensate for the loss of groundwater to industrial contamination in Albuquerque.

The state tried to show that the government’s past operations had contaminated the groundwater and would continue to do so for 50 years.  The demonstration rested on work performed by a dozen experts.  They assembled data from monitoring wells; mapped the older data (from 1989); and ran a groundwater model to predict where the contamination would spread.  They projected that contamination would affect a cubic mile of water, claiming it was worth billions to the city.

During World War 2, the United States government had operated manufacturing facilities far from most of the contamination.  The experts linked those facilities to the contamination by making a three-dimensional map that showed one contiguous plume in the groundwater.  They chose to use “kriging,” a statistical method of interpolating monitoring data across the region.  Kriging is often claimed to be “best” and “unbiased.”  For that reason it is a popular tool for mapping data where the interpretation may be disputed.   It can predict values at un-sampled locations:

“You want to establish a volume of contamination.  The only way that I know how to do that is to take real samples from groundwater, have them analyzed … to define the concentrations as a function of, say, three dimensional space.  Now, obviously because of economics and whatever we can't drill wells every one foot, can we?  So we drill wells where we can; we sample where we can; and then we have to have some kind of a method to fill in the blanks.  That's where the kriging comes in.” – Expert for the Plaintiffs.

Kriging capitalizes on Tobler’s First Law of Geography: “Everything is related to everything else, but near things are more related to each other.”  It builds a statistical model telling just how much the contamination at one location is related to that at other locations.  This works well when the statistical assumptions it makes are plausible.  Many practitioners of kriging are unaware of all these assumptions, leading to misapplication and poor predictions.  Dr. Huber therefore performed a systematic, detailed review of all the kriging steps to determine which assumptions were made and whether they were plausible.  

Using a careful analysis of the raw data, Dr. Huber was able to estimate their precision.  This precision turned out to be worse than was needed at a crucial step of the kriging.  By exploiting the analytical and visualization capabilities of a GIS to track down the implications, Dr. Huber discovered that the experts had completely invented a sequence of three artificial data points linking the DOJ facility to the contamination.  Without those imaginary “control points,” there would be no connection between the government’s past operations and the contaminated groundwater covered by the state’s claims.

Despite testimony from a renowned kriging expert supporting the Plaintiff’s work, the DOJ prevailed in a request for summary judgment.  They deserve the last word:

“After three years of litigation, the State of New Mexico dismissed claims against the United States seeking compensation for natural resource damages at the federal South Valley Superfund site near Albuquerque. … The court’s order ending the case against the United States provides no payment to the state or its private attorneys.
“On November 20, 2002, the court issued an order … ending the case against the United States “with prejudice,” precluding the state from refiling suit against the U.S. at a later date.” – DOJ press release, November 26, 2002.

South Valley is where much of AMC’s popular television series Breaking Bad was filmed.

Web links

http://www.justice.gov/opa/pr/2002/November/02_enrd_694.htm

http://caselaw.lp.findlaw.com/scripts/getcase.pl?navby=search&case=/data2/circs/10th/042191.html

http://www.bullivant.com/Tenth-Circuit-rejects-New-Mexicos-5-billion