In an enterprise of this scale, state social workers are bound to make mistakes when determining Medicaid eligibility and payment amounts. To assess the rate at which mistakes were made in enrolling patients and authorizing payments, each state selected roughly 300 to 800 randomly sampled cases, then audited each one to determine the Error Rate. Then, HHS took its own sample of the state sample – typically 30 to 80 cases – to determine its own error rate. This system of double sampling was intended to achieve the same accuracy as if HSS audited every Medicaid case in the state.
HHS assigned each state a target error rate for a specific period, and applied a pro-rated formula for assessing penalties. States would forfeit federal reimbursement based on how much their error rate exceeded the HSS target rate.
Because each state’s error rate could only be measured by sampling, a sampling error was built in. Even if over the years, a state’s error rate exactly equaled and never exceeded its target rate, it would still face substantial penalties due to unavoidable variations in the annual samples.
In perhaps half of the years, a state's error rate would be lower than the target, so no penalties are applied. But in the years when the error rate exceeded its target, millions of dollars in federal support payments would be withheld.
The Analysis & Inference principal observed that, for the states, the system was a “Heads I win, tails you lose” proposition. He dubbed this inequity the “Penalty Bias” since it penalized states for exceeding the target, but failed to reward them when they achieved a lower error rate.
The A&I principal also showed how the Penalty Bias could be calculated for each state.
Analysis & Inference's principal's creative insights helped the coalition of states represented by Covington & Burling mount a strong case. The HHS Departmental Appeals Board found the A&I principal's argument identifying a flaw in the quality control system persuasive, and overturned millions of dollars in penalties to the states.
“You will be pleased to learn that the Appeals Board has reversed the remaining sanctions,” wrote the lawyer for the states to the A&I principal. “I know you will enjoy reading the discussions of the unreasonably wide confidence intervals… and the failure to take into account…bias... You made a significant contribution to this victory.”
