How to exert greater management control over certain expenses for the Office of Accounting and Financial Management.

The project included drafting and revising statistical sampling plans, implementing the samples on a monthly or quarterly basis, and analyzing the results. Samples were grouped by expense amount, geographical region and division.

Improved financial management, including credit card usage, unused grant funds, accounts receivable, and travel expenses. The sampling plans and analyses were successfully defended before USGS outside auditor.

In a series of matters on behalf of the New York City Law Department, created and analyzed a massive real estate database, modeled market and sales values, and wrote expert reports to determine potential biases of alternative methods of valuing commercial real estate. Determined the validity of assumptions about lease lengths, turnover rates, and other issues affecting rents and property values.

New York City purchased over a hundred million dollars worth of new garbage trucks. Upon fielding the trucks the city expected to see more tons of garbage packed in than it found. The manufacturer offered to retro-fit the trucks with enhanced hyraulics to upgrade any deficiency in the vehicles' capacity. The question for the city was how to determine if the fix was sufficient.

A special Corporation Counsel asked Analysis & Inference if it could design a test to answer this question. In response, and in cooperation with a mechanical engineering consultant, we designed a field experiment in which a small number of retro-fitted trucks made runs and recorded their loads. The trucks were rotated among the different areas of the city on different days following a strict experimental protocol that had no truck repeat collection on the same day of the week in the same area during a week. Further, the protocol required uniform conditions of measurement of the tons packed in and for the operation of the vehicles to prevent other factors from influencing the results.

The statistical analysis of the tons packed out over the experimental runs showed a classic bell-shaped curve whose mean was comfortably above the spec, and whose 10th percentile was right at the spec. While the contract had not spelled out all the details of what such results would have to be to pass the trucks, the two parties agreed that the delivered vehicles were good.

Statisticians may be most useful when there are no statistics. Yes, you read that correctly. An example arose for Analysis & Inference in a case of alleged age discrimination. Statistics on rates of promotion by age and rank in an organization can typically be calculated from data collected routinely for personnel administration. By contrast, data on merit used in promotions are often absent or measured with substantial error.

However, so long as early promotion is credited with a positive correlation to ability, a simple model of promotion shows that--given no discrimination by age-rates of promotion in every rank have to be greater for younger versus older officers. Further, in this circumstance, raw rates of promotion by age do not estimate any parameter reasonably describing age discrimination.

This point was developed by statisticians at Analysis & Inference using several methodologies in the field of statistics. Application of the work was made to legal cases involving alleged age discrimination in a big city police force.

An Hispanic police officer alleged race discrimination by a Police Department in assignments made to the "Spare List." A Spare List refers to a list of officers who at any given time are available for assignments to cover for officers out for vacation or sickness or to police special events. Generally officers prefer to be on a regular beat. The expert for the plaintiff determined that a statistically significant difference between Hispanic and Caucasion officers in assignments to the Spare List existed.

However, there are a number of choices that can be made about how to measure "assignments to the Spare List." For example, do you count in any given period the numbers of times individual officers are assigned, the percent of their time spent on the list, when they are on the list, how long do they stay, and so forth. The selection of measures --in this case measures of discrimination -- is a less often noticed, but critical, step in determining what is going on.

In this case different measures resulted in sharp differences in the characterization of discrimination. In the end, in Federal District Court, the Department's use of the Spare List was not found to be discriminatory.

On behalf of the Arava Institute of Environmental Studies, advised on the design and sampling methodology for a broad-based survey of environmental education in middle and high schools. More than 7,000 students were surveyed in a sample that was stratified by size of town, income level, and other socio-economic variables. Performed weighted statistical analysis to project survey results to the population. Conclusions were presented before an Israeli Congressional Committee in a special session on the environment, the research resulted in publications in both English and Hebrew (Dr. Alan Salzberg, our CEO, was a co-author).

The U.S. Department of Health and Human Services (HHS) is responsible for oversight of federal aid given to states to support the major benefit programs of Temporary Assistance for Needy Families (formerly AFDC or welfare), Food Stamps, and Medicaid. As such HHS administers the largest quality control sampling scheme anywhere, in cooperation with the states. Findings of rates of error from state and federal samples of payments determined by state social workers are used to reduce pro rata amounts of federal aid to states that exceeded their target maximum error rates. These penalties have run to tens of millions of dollars.

In reviewing the program, Analysis & Inference identified important and previously unnoticed flaws in the statistical estimates of the penalties derived from the estimated error rates. The sampling methods themselves were well-designed by leading authorities in sample surveys, but their use in levying penalties was shown to be subject to important biases against states.

The HHS Departmental Appeals Board upheld state challenges to the penalties based in part on our statistical critique and testimony before the Board. The statisticians who worked on the project later published the statistical analysis, which applied modern Bayesian statistical methods, in the Journal of the American Statistical Association.

How to test for state regulatory commissions that a competitive environment exists in local telecommunications markets.

We created statistical sampling and data analysis plans to test and verify performance and incentive plans of telecommunications providers in a series of matters before state public service commissions in New York, Florida, Georgia, Michigan, Virginia, and Colorado. Dr. Salzberg was a co-inventor (Patent #6,636,585) of performance metrics used in this work. Executing the plans required processing and testing millions of yearly transactions. We implemented sampling plans and software programs to test the data integrity of vast "raw" datasets required to test and process these transactions and we reconstructed and replicated thousands of monthly statistical tests on the telecommunications data to ensure the tests were properly coded and implemented into the phone companies' systems.

Dr. Alan Salzberg, testified before the state public service commissions with the results of the statistical analyses and testing.

The New York City Public Schools were sued in a class action on behalf of pre-school children who were eligible for special education services subsidized by the State and the Federal government, over the timely delivery of services. The city asked Analysis & Inference to examine the timeliness of provision of services.

Substantial sample surveys of private providers of substantial sample surveys of private providers of special education services, and of city social services staffs that administer the program were created and executed. The project called for sorting out a complex intersection of causal influences on the timeliness of service provision. It also called on the use of concepts and methods from the field of quality control and improvement.

Counsel for the city credited the final Report written at Analysis & Inference as being very instrumental in achieving a settlement.

Were similar answers of defendant and another officer to questions on a promotion exam only coincidences, or did they establish cheating? We examined the question on behalf of the Police Department.

A statistical model of question-answering was developed, and the probability of the observed matching, or more, was determined both under hypotheses of no cheating and of cheating. We used a computer simulation of 100,000 people taking the test, and estimated the probabilities through the results.

In this case the probability of the observed matching, or more, under cheating was several orders of magnitude greater than under no cheating.

A state ran afoul of a federal audit, which disallowed a portion of federal aid for one of the state's human services programs on the ground that the customary sample of social workers' time uses for a period had not been collected during some months' time related to a transition in the state department.

Forecasting the missing time from past years' data to the missing period, and also backcasting from data that became available subsequently, A&I showed how the missing records could be estimated within a reasonable degree of precision.

A senior statistician at Analysis & Inference testified, before the federal department's Departmental Appeals Board, to the commonly accepted uses of forecasting in financial fields, and showed how methods of time series analysis could be reliably applied to the given data. The federal position was that no substitute could be made for the specific estimates obtained from a sample of time uses. The Board did not decide for the state. However, the Assistant Attorney General said: "Notwithstanding the result, I appreciate the excellent job you did for the State."

The New York City Commissioner of Human Resources was sued in a class action on behalf of aliens living in the City who were eligible for social services, which they alleged were being denied to them unlawfully.

The City asked Dr. Fairley to aid its lawyers in negotiating a settlement with plaintiffs. Plaintiffs had proposed a periodic sample of the actions of staff of City social services programs in accepting applicants and making decisions about benefits for persons already under care. Dr. Fairley was asked to design the sample, in cooperation with the Human Resources Administration.

He participated in the creation of a stipulation between the parties that spelled out the criteria for sampling and many of the features of the sampling that both parties agreed upon.

New York City contracted with Brink's Inc. to collect coins from parking meters. After city officials were informed that some of Brinks' employees had been stealing, six were arrested and convicted of larceny. The city filed a complaint against Brink's for negligence in supervising its employees and for breaching the contract. The case came to trial in the United States District Court for the Southern District of New York. (The City had no direct knowledge of the exact amount of money that had been stolen.)

Analysis & Inference's assignment was to estimate the amount stolen by comparing Brinks' actual collections with estimates of how much should have been collected. Those estimates, generated by studying revenues of the subsequent contractor for matched months and boroughs and regression modeling of expected revenues versus actual, were successfully presented before a jury in the District Court.

In upholding the decision for the city, Judge Weinfeld of the United States Court of Appeals quoted the testimony of Analysis & Inference against criticisms that the statistical estimate was not precise: "Dr. Fairley said the purpose of [statistical analysis] is: what is the most reasonable estimate of the difference and what [can you] most reasonably attribute the difference? That's the best you can do." Final judgment against Brink's was $2.5 million of which $1 million was compensatory and $1.5 was punitive.

In a subsequent case of stolen parking collections in the City, following a presentation to the contractor the City considered responsible, Dr. Fairley showed that party a simple graphical analysis of the amount of the theft. The analysis followed that used by Dr. Fairley so effectively in the previous case that had gone to trial. The contractor conceded their responsibility and settled the case soon after.