7.3.3. How can we determine whether two processes produce the same proportion of defectives?

7. Product and Process Comparisons
7.3. Comparisons based on data from two processes

7.3.3. How can we determine whether two processes produce the same proportion of defectives?

Case 1: Large Samples (Normal Approximation to Binomial)

The hypothesis of equal proportions can be tested using a z statistic

If the samples are reasonably large we can use the normal approximation to the binomial to develop a test similar to testing whether two normal means are equal.

Let sample 1 have x₁ defects out of n₁ and sample 2 have x₂ defects out of n₂. Calculate the proportion of defects for each sample and the z statistic below:

z = (phat1 - phat2)/SQRT(phat*(1-phat)*((1/n1) + (1/n2))

where

phat = (n1*phat1 + n2*phat2)/(n1+n2) = (x1 + x2)/(n1+n2)

Compare z to the normal z table value for a 2-sided test. For a one sided test, assuming the alternative hypothesis is p₁ > p₂, compare z to the normal z table value. If the alternative hypothesis is p₁ < p₂, compare z to -z.

Case 2: An Exact Test for Small Samples

The Fisher Exact Probability test is an excellent choice for small samples

The Fisher Exact Probability Test is an excellent nonparametric technique for analyzing discrete data (either nominal or ordinal), when the two independent samples are small in size. It is used when the results from two independent random samples fall into one or the other of two mutually exclusive classes (i.e., defect versus good, or successes vs failures).

Example of a 2x2 contingency table

In other words, every subject in each group has one of two possible scores. These scores are represented by frequencies in a 2x2 contingency table. The following discussion, using a 2x2 contingency table, illustrates how the test operates.

We are working with two independent groups, such as experiments and controls, males and females, the Chicago Bulls and the New York Knicks, etc.

- + Total

Group I A B A+B

Group II C D C+D

Total A+C B+D N

The column headings, here arbitrarily indicated as plus and minus, may be of any two classifications, such as: above and below the median, passed and failed, Democrat and Republican, agree and disagree, etc.

Determine whether two groups differ in the proportion with which they fall into two classifications

Fisher's test determines whether the two groups differ in the proportion with which they fall into the two classifications. For the table above, the test would determine whether Group I and Group II differ significantly in the proportion of plusses and minuses attributed to them.

The method proceeds as follows:

The exact probability of observing a particular set of frequencies in a 2 × 2 table, when the marginal totals are regarded as fixed, is given by the hypergeometric distribution

p = (A+C A)/(B+D B)/(N A+B)
= {(A+C)!*(B+D)!/(A!*C!*B!*D!)}/{N!/[(A+B)!*(C+D)!]}
= (A+B)!*(C+D)!*(A+C)!*(B+D)!/[N!*A!*B!*C!*D!]

But the test does not just look at the observed case. If needed, it also computes the probability of more extreme outcomes, with the same marginal totals. By "more extreme", we mean relative to the null hypothesis of equal proportions.

Example of Fisher's test

This will become clear in the next illustrative example. Consider the following set of 2 x 2 contingency tables:

Observed Data More extreme outcomes with same marginals

(a) (b) (c)

2 5 7

3 2 5

5 7 12

1 6 7

4 1 5

5 7 12

0 7 7

5 0 5

5 7 12

Table (a) shows the observed frequencies and tables (b) and (c) show the two more extreme distributions of frequencies that could occur with the same marginal totals 7, 5. Given the observed data in table (a) , we wish to test the null hypothesis at, say, alpha = .05.

Applying the previous formula to tables (a), (b), and (c), we obtain

p(a) = 7!*5!*5!*7!/[12!*2!*5!*3!*2!] = 0.26515

p(b) = 7!*5!*5!*7!/[12!*1!*6!*4!*1!] = 0.04419

p(c) = 7!*5!*5!*7!/[12!*0!*7!*5!*0!] = 0.00216

The probability associated with the occurrence of values as extreme as the observed results under H₀ is given by adding these three p's:

.26515 + .04419 + .00126 = .31060

So p = .31060 is the probability that we get from Fisher's test. Since .31060 is larger than alpha , we cannot reject the null hypothesis.

Tocher's Modification

Tocher's modification makes Fisher's test less conservative

Tocher (1950) showed that a slight modification of the Fisher test makes it a more useful test. Tocher starts by isolating the probability of all cases more extreme than the observed one. In this example that is

p_b + p_c = .04419 + .00126 = .04545

Now, if this probability is larger than alpha , we cannot reject H_o. But if this probability is less than alpha , while the probability that we got from Fisher's test is greater than alpha (as is the case in our example) then Tocher advises to compute the following ratio:

(alpha - P(more extreme case)/p(observed alone)

For the data in the example, that would be

(alpha - (p(b) + p(c))/p(a) = (.05 - .04545)/.26515 = .0172

Now we go to a table of random numbers and at random draw a number between 0 and 1. If this random number is smaller than the ratio above of .0172, we reject H₀. If it is larger we cannot reject H₀. This added small probability of rejecting H₀ brings the test procedure Type I error (i.e., alpha value) to exactly .05 and makes the Fisher test less conservative.

The test is a one-tailed test. For a two-tailed test, the value of p obtained from the formula must be doubled.

A difficulty with the Tocher procedure is that someone else analyzing the same data would draw a different random number and possibly make a different decision about the validity of H₀.