1. Decreasing the confidence in the estimate -- an undesirable strategy since confidence relates to the chance of drawing the wrong conclusion (i.e., increases the Type II error).
2. Reducing the standard deviation -- something we cannot do since it is usually a static property of the population.
3. Increasing the sample size -- this provides more information for a better decision.

You might like to use Descriptive Statistics JavaScript to check your computations, and Sample Size Determination JavaScript at the design stage of your statistical investigation in decision making with specific subjective requirements.

Further Reading
Levy P., and S. Lemeshow, Sampling of Populations: Methods and Applications, Wiley, 1999.

---

Bias Reduction Techniques: Bootstrapping and Jackknifing

Some inferencial statistical techniques do not require distributional assumptions about the statistics involved. These modern non-parametric methods use large amounts of computation to explore the empirical variability of a statistic, rather than making a priori assumptions about this variability, as is done in the traditional parametric t- and z- tests.

Bootstrapping: Bootstrapping method is to obtain an estimate by combining estimators to each of many sub-samples of a data set. Often M randomly drawn samples of T observations are drawn from the original data set of size n with replacement, where T is less n.

Jackknife Estimator: A jackknife estimator creates a series of estimate, from a single data set by generating that statistic repeatedly on the data set leaving one data value out each time. This produces a mean estimate of the parameter and a standard deviation of the estimates of the parameter.

Monte Carlo simulation allows for the evaluation of the behavior of a statistic when its mathematical analysis is intractable. Bootstrapping and jackknifing allow inferences to be made from a sample when traditional parametric inference fails. These techniques are especially useful to deal with statistical problems, such as small sample size, statistics with no well-developed distributional theory, and parametric inference condition violations. Both are computer intensive. Bootstrapping means you take repeated samples from a sample and then make statements about a population. Bootstrapping entails sampling-with-replacement from a sample. Jackknifing involves systematically doing n steps, of omitting 1 case from a sample at a time, or, more generally, n/k steps of omitting k cases; computations that compare "included" vs. "omitted" can be used (especially) to reduce the bias of estimation. Both have applications in reducing bias in estimations.

Resampling -- including the bootstrap, permutation, and other non-parametric tests -- is a method for hypothesis testing, confidence limits, and other applied problems in statistics and probability. It involves no formulas or tables.

Following the first publication of the general technique (and the bootstrap) in 1969 by Julian Simon and subsequent independent development by Bradley Efron, resampling has become an alternative approach for testing hypotheses.

There are other findings: "The bootstrap started out as a good notion in that it presented, in theory, an elegant statistical procedure that was free of distributional conditions. In practice the bootstrap technique doesn't work very well, and the attempts to modify it make it more complicated and more confusing than the parametric procedures that it was meant to replace."

While resampling techniques may reduce the bias, they achieve this at the expense of increase in variance. The two major concerns are:

1. The loss in accuracy of the estimate as measured by variance can be very large.
2. The dimension of the data affects drastically the quality of the samples and therefore the estimates.

Further Readings:
Young G., Bootstrap: More than a Stab in the Dark?, Statistical Science, l9, 382-395, 1994. Provides the pros and cons on the bootstrap methods.
Yatracos Y., Assessing the quality of bootstrap samples and of the bootstrap estimates obtained with finite resampling, Statistics and Probability Letters, 59, 281-292, 2002.

---

Prediction Intervals

In many application of business statistics, such as forecasting, we are interested in construction of a statistical interval for random variable, rather than a parameter of a population distribution.

• The Tchebysheff's inequality is often used to put bounds on the probability that a proportion of random variable X will be within k > 1 standard deviation of the mean m for any probability distribution. In other words:
  P [|X - m| ³ k s] £ 1/k²,   for any k greater than 1

  The symmetric property of Tchebysheff's inequality is useful; e.g., in constructing control limits in the quality control process. However, the limits are very conservative due to lack of knowledge about the underlying distribution.

• The above bounds can be improved (i.e., becomes tighter) if we have some knowledge about the population distribution. For example, if the population is homogeneous; that is, its distribution is unimodal; then,
  P [|X - m| ³ k s] £ 1/(2.25k²),   for any k greater than 1.

  The above inequality is known as the Camp-Meidell inequality.

• Now, let X be a random variable distributed normally with estimated mean and standard deviation S, then a prediction interval for the sample mean with 100(1- a)% confidence level is:

  ± t_a/2 ´ S ´ (1+1/n)^1/2.

  This is the range of a random variable with 100(1- a)% confidence, using t-table. Relaxing the normality condition for sample-mean prediction interval, requires a large sample size, say n over 30.

Further Readings:
Grant E., and R. Leavenworth, Statistical Quality Control, McGraw-Hill, 1996.
Ryan T., Statistical Methods for Quality Improvement, John Wiley & Sons, 2000. A very good book for a starter.

---

What Is a Standard Error?

For statistical inference, namely statistical testing and estimation, one needs to estimate the population's parameter(s). Estimation involves the determination, with a possible error due to sampling, of the unknown value of a population parameter, such as the proportion having a specific attribute or the average value m of some numerical measurement. To express the accuracy of the estimates of population characteristics, one must also compute the standard errors of the estimates. These are measures of accuracy that determine the possible errors arising from the fact that the estimates are based on random samples from the entire population, and not on a complete population census.

Standard error is a statistic indicating the accuracy of an estimate. That is, it tells us to assess how different the estimate (such as ) is from the population parameter (such as m). It is therefore, the standard deviation of a sampling distribution of the estimator such as . The following is a collection of standard errors for the widely used statistics:

• Standard Error for the Mean is: S/n^½.
  As one expects, the standard error decreases as the sample size increases. However the standard deviation of the estimate decreases by a factor of n^½ not n. For example, if you wish to reduce the error by 50%, the sample size must be 4 times n, which is expensive. Therefore, as an alternative to increasing sample size, one may reduce the error by obtaining "quality" data that provide a more accurate estimate.

• For a finite population of size N, the standard error of the sample mean of size n, is:

  S ´ [(N -n)/(nN)]^½.

• Standard Error for the Multiplication of Two Independent Means ₁ ´ ₂ is:

  {₁ S₂²/n₂ + ₂ S₁²/n₁}^½.

• Standard Error for Two Dependent Means ₁ ± ₂ is:

  {S₁²/n₁ + S₂²/n₂ + 2 r ´ [(S₁²/n₁)(S₂²/n₂)]^½}^½.

• Standard Error for the Proportion P is:

  [P(1-P)/n]^½

• Standard Error for P₁ ± P₂, Two Dependent Proportions is:

  {[P₁ + P₂ - (P₁-P₂)²] / n}^½.

• Standard Error of the Proportion (P) from a finite population is:

  [P(1-P)(N -n)/(nN)]^½.

  The last two formulas for finite population are frequently used when we wish to compare a sub-sample of size n with a larger sample of size N, which contains the sub-sample. In such a comparison, it would be wrong to treat the two samples "as if" there were two independent samples. For example, in comparing the two means one may use the t-statistic but with the standard error:

  S_N [(N -n)/(nN)]^½

  as its denominator. Similar treatment is needed for proportions.

• Standard Error of the Slope (m) in Linear Regression is

  S_res / S_xx^½, where S_res is the residual' standard deviation.

• Standard Error of the Intercept (b) in Linear Regression is:

  S_res[(S_xx + n ´ ²) /(n ´ S_xx] ^½.

• Standard Error of the Predicted Value using a Linear Regression is:

  S_y(1 - r²)^½.

  The term (1 - r²)^½ is called the coefficient of alienation. Therefore if r = 0, the error of prediction is S_y as expected.

• Standard Error of the Linear Regression is:

  S_y (1 - r²)^½.

  Note that if r = 0, then the standard error reaches its maximum possible value, which is standard deviation in Y.

Stability of an estimator: An estimator is stable if, by taking two different samples of the same size, they produce two estimates having "small" absolute difference. The stability of an estimator is measured by its reliability:

Reliability of an estimator = 1 / (its standard error)²

The larger the standard error, the less reliable is the estimate. Reliability of estimators is often used to select the "best" estimator among all unbiased estimators.

---

Sample Size Determination

At the planning stage of a statistical investigation, the question of sample size (n) is critical. This is an important question therefore it should not be taken lightly. To take a larger sample than is needed to achieve the desired results is wasteful of resources, whereas very small samples often lead to what are no practical use of making good decisions. The main objective is to obtain both a desirable accuracy and a desirable confidence level with minimum cost.

Students sometimes ask me, what fraction of the population do you need for good estimation? I answer, "It's irrelevant; accuracy is determined by sample size." This answer has to be modified if the sample is a sizable fraction of the population.

The confidence level of conclusions drawn from a set of data depends on the size of the data set. The larger the sample, the higher is the associated confidence. However, larger samples also require more effort and resources. Thus, your goal must be to find the smallest sample size that will provide the desirable confidence.

For an item scored 0 or 1, for no or yes, the standard error (SE) of the estimated proportion p, based on your random sample observations, is given by:

SE = [p(1-p)/n]^1/2

where p is the proportion obtaining a score of 1, and n is the sample size. This SE is the standard deviation of the range of possible estimate values.

The SE is at its maximum when p = 0.5, therefore the worst case scenario occurs when 50% are yes, and 50% are no.

Under this extreme condition, the sample size, n, can then be expressed as the largest integer less than or equal to:

n = 0.25/SE²

To have some notion of the sample size, for example for SE to be 0.01 (i.e. 1%), a sample size of 2500 will be needed; 2%, 625; 3%, 278; 4%, 156, 5%, 100.

Note, incidentally, that as long as the sample is a small fraction of the total population, the actual size of the population is entirely irrelevant for the purposes of this calculation.

Pilot Studies: When the needed estimates for sample size calculation is not available from an existing database, a pilot study is needed for adequate estimation with a given precision. A pilot, or preliminary, sample must be drawn from the population, and the statistics computed from this sample are used in determination of the sample size. Observations used in the pilot sample may be counted as part of the final sample, so that the computed sample size minus the pilot sample size is the number of observations needed to satisfy the total sample size requirement.

Sample Size with Acceptable Absolute Precision: The following present the widely used method for determining the sample size required for estimating a population mean and proportion.

Let us suppose we want an interval that extends d unit on either side of the estimator. We can write

d = Absolute Precision = (reliability coefficient) ´ (standard error) = Z_a/2 ´ (S/n^1/2)

Suppose, based on a pilot sample of size n, the estimated proportion is p, then the required sample size with the absolute error size not exceeding d, with 1- a confidence is:

[t² n p(1-p)] / [t² p(1-p) - d² (n-1)],

where t = t _a/2 being the value taken from the t-table with parameter d.f. = n = n-1, corresponding to the desired 1- a confidence interval.

For large pilot sample sizes (n), say over 30, the simplest sample size determinate is:

[(Z_a/2)² S²] / d²     for the Mean m

[(Z_a/2)² p(1-p)] / d²     for the proportion,

where d is the desirable margin of error (i.e., the absolute error), which is the half-length of the confidence interval with 100(1- a)% confidence interval.

Sample Size with Acceptable Type I and Type II Errors: One may use the following sample size determinate, which is based on the size of type I and Type II errors:

2(Z_a/2 + Z_b/2)²S²/d²,

where a and b are the desirable type I, and type II errors, respectively. S² is the variance obtained from the pilot run, and d is the difference between the null and alternative (m₀ -m_a).

Sample Size with Acceptable Relative Precision: You may use the following sample size determinate for a desirable relative error D in %, which requires an estimate of the coefficient of variation (CV in %) from a pilot sample with size over 30:

[(Z_a/2)² (C.V.)²] / D² 

Sample Size Based on the Null and an Alternative: One may use power of the test to determine the sample size. The functional relation of the power and the sample size is known as the operating characteristic curve. On this curve, as sample size increases, the power function increases rapidly. Let d be such that:

m_a = m₀ + d

is an alternative to represent departure from the null hypothesis. We wish to be reasonably confident to find evidence against the null, if in fact the particular alternative holds. That is, the type error b, is the probability of failing to find evidence at least at level of a, when the alternative holds. This implies

Required sample size = (z₁ + z₂) S²/ d²

Where: z₁ = |mean - m₀|/ SE, z₂ = |mean - m_a|/ SE, the mean is the current estimate for m, and S is the current estimate for s.

All of the above sample size determinates could also be used for estimating the mean of any unimodal population, with discrete or continuous random variables, provided the pilot run size (n) is larger than (say) 30.

In estimating the sample size, when the standard deviation is not known, instead of S² one may use 1/4 of the range for sample size over 30 as a "good" estimate for the standard deviation. It is a good practice to compare the result with IQR/1.349.

One may extend the sample size determination to other useful statistics, such as correlation coefficient (r) based on acceptable Type I and Type II errors:

2 + [(Z_a/2 + Z_b/2( 1- r²) ^½)/r] ²

provided r is not equal to -1, 0, or 1.

The aim of applying any one of the above sample size determinates is at improving your pilot estimates at feasible costs.

You might like to use Sample Size Determination JavaScript to check your computations.

Further Reading:
Kish L., Survey Sampling, Wiley, 1995.
Murphy K., and B. Myors, Statistical Power Analysis, L. Erlbaum Associates, 1998. Provides a simple and general sample size determination for hypothesis tests.

---

Revising the Expected Value and the Variance

Averaging Variances: What is the mean variance of k variances without regard to differences in their sample sizes? The answer is simply:

Average of Variances = [SS_i²] / k

However, what is the variance of all k groups combined? The answer must consider the sample size n_i of the ith group:

Combined Group Variance = S n_i[S_i² + d_i²]/N,

where d_i = mean_i - grand mean, and N = S n_i, for all i = 1, 2, .., k.

Notice that the above formula allows us to split up the total variance into its two component parts. This splitting process permits us to determine the extent to which the overall variation is inflated by the difference between group means. What the variation would be if all groups had the same mean? ANOVA is a well-known application of this concept where the equality of several means is tested.

Subjective Mean and Variance: In many applications, we saw how to make decisions based on objective data; however, an informative decision-maker might be able to combine his/her subjective input and the two sources of information.

Application: Suppose the following information is available from two independent sources:

Revising the Expected Value and the Variance
Estimate Source	Expected value	Variance
Sales manager	m1 = 110	s12 = 100
Market survey	m2 = 70	s22 = 49

The combined expected value is:

[m₁/s₁² + m₂/s₂² ] / [1/s₁² + 1/s₂²]

The combined variance is:

2 / [1/s₁² + 1/s₂²]

For our application, using the above tabular information, the combined estimate of expected sales is 83.15 units with combined variance of 65.77.

You might like to use Revising the Mean and Variance JavaScript in performing some numerical experimentation. You may apply it for validating the above example and for a deeper understanding of the concept where more than two sources of information are to be combined.

---

Subjective Assessment of Several Estimates Based on Relative Precision

In many cases, we may wish to compare several estimates of the same parameter. The simplest approach is to measure the closeness among the estimates in an attempt to determine that at least one of the estimates is more than r times the parameter away from the parameter, where r is a subjective, non-negative number less than one.

You might like to use Subjective Assessment of Estimates JavaScript to isolate any inaccurate estimate. By repeating the same process you might be able to remove all inaccurate estimates.

Further Reading:
Tsao H. and T. Wright, On the maximum ratio: A tool for assisting inaccuracy assessment, The American Statistician, 37(4), 1983.

---

Bayesian Statistical Inference: An Introduction

Statistical inference describes the procedures by which we use the observed data to draw conclusions about the population from which the data came or about the process by which the data were generated. Our assumption is that there is an unknown process that generates the data we have and that this process can be described by a probability distribution, which, in turn, can be characterized by some unknown parameters. For instance, for a normal distribution the unknown parameters are m and s².

Broadly speaking, statistical inference can be classified under two headings: classical inference and Bayesian inference. Classical statistical inference is based on two premises:

1. The sample data constitute the only relevant information.
2. The construction and assessment of the different procedures for inference are based on long-run behavior under essentially similar circumstances.

In Bayesian inference we combine sample information with prior information. Suppose that we draw a random sample x1, x2,....xn of size n from a normal population.

In statistical inference we take the sample mean as our estimate of m. Its variance is s² / n. The inverse of this variance is known as the sample precision. Thus the sample precision is n / s².

In the Bayesian inference we have prior information on m. This is expressed in terms of a probability distribution known as the prior distribution. Suppose that the prior distribution is normal with mean m₀ and variance s₀², that is, precision 1/ s₀². We now combine this with the sample information to obtain what is known as the posterior distribution of µ. This distribution can be shown to be normal. Its mean is a weighted average of the sample mean and the prior mean, weighted by the sample precision and prior precision, respectively. Thus,

Posterior mean = (W₁ + W₂ m₀) / (W₁ + W₂)

Posterior variance = 1 / (W₁ + W₂)

where

W1 = Sample precision = n/S², and W1 = Prior precision = n/s₀²

Also, the precision (or inverse of the variance) of the posterior distribution of m is W₁ + W₂, that is, the sum of the sample precision and prior precision.

The posterior mean will lie between the sample mean and the prior mean. The posterior variance will be less than both the sample and prior variances.

In this Web site do not discuss Bayesian inference because this would take us into a lot more detail than we intend to cover. However, the basic notion of combining the sample mean and prior mean in inverse proportion to their variances will be of interest while being useful.

You may like using the Bayesian Statistical Inference JavaScript for checking your computation and performing some experiment.

---

Managing the Producer's or the Consumer's Risk

The logic behind a statistical test of hypothesis is similar to the following logic. Draw two lines on a paper and determine whether they are of different lengths. You compare them and say, "Well, certainly they are not equal. Therefore they must be of different lengths. By rejecting equality, that is, the null hypothesis, you assert that there is a difference.

The power of a statistical test is best explained by the overview of the Type I and Type II errors. The following matrix shows the basic representation of these errors.

As indicated in the above matrix a Type-I error occurs when, based on your data, you reject the null hypothesis when in fact it is true. The probability of a type-I error is the level of significance of the test of hypothesis and is denoted by a .

Type-I error is often called the producer's risk that consumers reject a good product or service indicated by the null hypothesis. That is, a producer introduces a good product, in doing so, he or she take a risk that consumer will reject it.

A type II error occurs when you do not reject the null hypothesis when it is in fact false. The probability of a type-II error is denoted by b . The quantity 1 - b is known as the Power of a Test. A Type-II error can be evaluated for any specific alternative hypotheses stated in the form "Not Equal to" as a competing hypothesis.

Type-II error is often called the consumer's risk for not rejecting possibly a worthless product or service indicated by the null hypothesis.

Students often raise questions, such as what are the 'right' confidence intervals, and why do most people use the 95% level? The answer is that the decision-maker must consider both the Type I and II errors and work out the best tradeoff. Ideally one wishes to reduce the probability of making these types of error; however, for a fixed sample size, we cannot reduce one type of error without at the same time increasing the probability of another type of error. Nevertheless, to reduce the probabilities of both types of error simultaneously is to increase the sample size. That is, by having more information one makes a better decision.

The following example highlights this concept. A electronics firm, Big Z, manufactures and sells a component part to a radio manufacturer, Big Y. Big Z consistently maintain a component part failure rate of 10% per 1000 parts produced. Here Big Z is the producer and Big Y is the consumer. Big Y, for reasons of practicality, will test sample of 10 parts out of lots of 1000. Big Y will adopt one of two rules regarding lot acceptance:

• Rule 1: Accept lots with one or fewer defectives; therefore, a lot has either 0 defective or 1 defective.
• Rule 2: Accept lots with two or fewer defectives; therefore, a lot has either 0,1, or 2 defective(s).

On the basis of the binomial distribution, the P(0 or 1) is 0.7367. This means that, with a defective rate of 0.10, the Big Y will accept 74% of tested lots and will reject 26% of the lots even though they are good lots. The 26% is the producer's risk or the a level. This a level is analogous to a Type I error -- rejecting a true null. Or, in other words, rejecting a good lot. In this example, for illustration purposes, the lot represents a null hypothesis. The rejected lot goes back to the producer; hence, producer's risk. If Big Y is to take rule 2, then the producer's risk decreases. The P(0 or, or 1, or 2) is 0.9298 therefore, Big Y will accept 93% of all tested lots, and 7% will be rejected, even though the lot is acceptable. The primary reason for this is that, although the probability of defective is 0.10, the Big Y through rule 2 allows for a higher defective acceptance rate. Big Y increases its own risk (consumer's risk), as stated previously.

Making Good Decision: Given that there is a relevant profit (which could be negative) for the outcome of your decision, and a prior probability (before testing) for the null hypothesis to be true, the objective is to make a good decision. Let us denote the profits for each cell in the decision table as $a, $b, $c and $d (column-wise), respectively. The expectation of profit is [aa + (1-a)b], and + [(1-b)c + bd], depending whether the null is true.

Now having a prior (i.e., before testing) subjective probability of p that the null is true, then the expected profit of your decision is:

Net Profit = [aa + (1-a)b]p + [(1-b)c + bd](1-p) - Sampling cost

A good decision makes this profit as large as possible. To this end, we must suitably choose the sample size and all other factors in the above profit function.

Note that, since we are using a subjective probability expressing the strength of belief assessment of the truthfulness of the null hypothesis, it is called a Bayesian Approach to statistical decision making, which is a standard approach in decision theory.

You might like to use the Subjectivity in Hypothesis Testing JavaScript in performing some numerical experimentation for validating the above assertions for a deeper understanding.

Further Reading:
Cochran W., Planning and Analysis of Observational Studies, Wiley, 1983.

---

Hypothesis Testing: Rejecting a Claim

To perform a hypothesis test, one must be very specific about the test one wishes to perform. The null hypothesis must be clearly stated, and the data must be collected in a repeatable manner. If there is any subjectivity, the results are technically not valid. All of the analyses, including the sample size, significance level, the time, and the budget, must be planned in advance, or else the user runs the risk of "data diving".

Hypothesis testing is mathematical proof by contradiction. For example, for a Student's t test comparing two groups, we assume that the two groups come from the same population (same means, standard deviations, and in general same distributions). Then we do our best to prove that this assumption is false. Rejecting H₀ means either H₀ is false, or a rare event as has occurred.

The real question is in statistics not whether a null hypothesis is correct, but whether it is close enough to be used as an approximation.

Test of Hypotheses
Click on the image to enlarge it and THEN print it

In most statistical tests concerning m, we start by assuming the s², and the higher moments, such as skewness and kurtosis, are equal. Then, we hypothesize that the a's are equal wich is null hypothesis.

The "null" often suggests no difference between group means, or no relationship between quantitative variables, and so on.

Then we test with a calculated t-value. For simplicity, suppose we have a two-sided test. If the calculated t is close to 0, we say "it is good", as we expected. If the calculated t is far from 0, we say, "the chance of getting this value of t, given my assumption that the populations are statistically the same, is so small that I will not believe the assumption. We will say that the populations are not equal; specifically the means are not equal."

As an example, sketch a normal distribution with mean 1 - 2 and standard deviation s. If the null hypothesis is true, then the mean is 0. We calculate the 't' value, as per the equation. We look up a "critical" value of t. The probability of calculating a t value more extreme ( + or - ) than this, given that the null hypothesis is true, is equal or less than the a risk we used in pulling the critical value from the table. Mark the calculated t, and critical t (both sides) on the sketch of the distribution. Now, if the calculated t is more extreme than the critical value, we say, "the chance of getting this t, by shear chance, when the null hypothesis is true, is so small that I would rather say the null hypothesis is false, and accept the alternative, that the means are not equal." When the calculated value is less extreme than the calculated value, we say, "I could get this value of t by shear chance. I cannot detect a difference in the means of the two groups at the a significance level."

In this test, we need (among others) the condition that the population variances (i.e., treatment impacts on central tendency but not variability) are equal. However, this test is robust to violations of that condition if n's are large and almost the same size. A counter example would be to try a t-test between (11, 12, 13) and (20, 30, 40). The pooled and unpooled tests both give t statistics of 3.10, but the degrees of freedom are different: d.f. = 4 (for pooled) or d.f. about 2 (for unpooled). Consequently the pooled test gives p = 0.036 and the unpooled p = 0.088. We could go down to n = 2 and get something still more extreme.

You might like to use Online Statistical Computation, Testing the Mean, and Testing the Variance in performing more of these tests.

You might need to use Sample Size Determination JavaScript at the design stage of your statistical investigation in decision making with specific subjective requirements.

---

Classical Approach to Testing Hypotheses

In this treatment there are two parties: One party (or a person) proposes the null hypothesis (the claim). Another party proposes an alternative hypothesis. A significance level a and a sample size n are agreed upon by both parties. The next step is to compute the relevant statistic based on the null hypothesis and the random sample of size n. Finally, one determines the rejection region. The conclusion based on this approach is as follows:

If the computed statistic falls within the rejection region, then Reject the null hypothesis; otherwise Do Not Reject the null hypothesis (the claim).

You may ask: How do you determine the critical value (such as z-value) for the rejection interval for one and two-tailed hypotheses?. What is the rule?

First, you have to choose a significance level a. Knowing that the null hypothesis is always in "equality" form then, the alternative hypothesis has one of the three possible forms: "greater-than", "less-than", or "not equal to". The first two forms correspond to a one-tail hypothesis while the last one corresponds to a two-tail hypothesis.

• If your alternative is in the form of "greater-than", then z is the value that gives you an area to the right tail of the distribution that is equal to a.

• If your alternative is in the form of "less-than", then z is the value that gives you an area to the left tail of the distribution that is equal to a.

• If your alternative is in the form of "not equal to", then there are two z values, one positive and the other negative. The positive z is the value that gives you an a/2 area to the right tail of the distribution. While, the negative z is the value that gives you an a/2 area to the left tail of the distribution.

The above rule can be generalized and implemented for determining the critical value for any test of hypothesis, you must first master reading the statistical tables, because, as you see, not all tables in your textbook are presented in the same format.

---

The Meaning and Interpretation of P-values (what the data say?)

The p-value, which directly depends on a given sample attempts to provide a measure of the strength of the results of a test for the null hypothesis, in contrast to a simple reject or do not reject in the classical approach to the test of hypotheses. If the null hypothesis is true, and if the chance of random variation is the only reason for sample differences, then the p-value is a quantitative measure to feed into the decision-making process as evidence. The following table provides a reasonable interpretation of p-values:

P-value	Interpretation
P < 0.01	very strong evidence against H0
0.01£ P < 0.05	moderate evidence against H0
0.05 £ P < 0.10	suggestive evidence against H0
0.10 £ P	little or no real evidences against H0

This interpretation is widely accepted, and many scientific journals routinely publish papers using this interpretation for the result of a test of hypothesis.

For the fixed-sample size, when the number of realizations is decided in advance, the distribution of p is uniform, assuming the null hypothesis is true. We would express this as P(p £ x) = x. That means the criterion of p £ 0.05 achieves a of 0.05.

Understand that the distribution of p-values under null hypothesis H₀ is uniform, and thus does not depend on a particular form of the statistical test. In a statistical hypothesis test, the P value is the probability of observing a test statistic at least as extreme as the value actually observed, assuming that the null hypothesis is true. The value of p is defined with respect to a distribution. Therefore, we could call it "model-distribution hypothesis" rather than "the null hypothesis".

In short, it simply means that, if the null had been true, the p-value is the probability against the null in that case. The p-value is determined by the observed value; however, this makes it difficult to even state the inverse of p.

Finally, since the p-values are random variables, one cannot compare several p-values for any statistical conclusions (nor order them). This is a common mistake many people do, therefore, the above table is not intended for such a comparison.

You might like to use The P-values for the Popular Distributions JavaScript.

Further Readings:
Arsham H., Kuiper's P-value as a Measuring Tool and Decision Procedure for the Goodness-of-fit Test, Journal of Applied Statistics, Vol. 15, No.3, 131-135, 1988.
Good Ph.., Resembling Methods: A Practical Guide to Data Analysis, Springer Verlag, 1999.

---

Blending the Classical and the P-value Based Approaches in Test of Hypotheses

A p-value is a measure of how much evidence you have against the null hypothesis. Notice that the null hypothesis is always in = form, and does not contain any forms of inequalities. The smaller the p-value, the more evidence you have. In this setting, the p-value is based on the hull hypothesis and has nothing to do with an alternative hypothesis and therefore with the rejection region. In recent years, some authors try to use the mixture of the classical and the p-value approaches. It is based on the critical value obtained from given a, the computed statistics and the p-value. This is a blend of two different schools of thought. In this setting, some textbooks compare the p-value with the significance level to make decisions on a given test of hypothesis. The larger the p-value is when compared with a (in one-sided alternative hypothesis, and a/2 for the two sided alternative hypotheses), the less evidence we have for rejecting the null hypothesis. In such a comparison, if the p-value is less than some threshold (usually 0.05, sometimes a bit larger like 0.1 or a bit smaller like 0.01) then you reject the null hypothesis. The following deal with such a combined approach.

Use of P-value and a: In this setting, we must also consider the alternative hypothesis in drawing the rejection region. There is only one p-value to compare with a (or a/2). Know that, for any test of hypothesis, there is only one p-value. The following outlines the computation of the p-value and the decision process involved in a given test of hypothesis:

1. P-value for One-sided Alternative Hypotheses: The p-value is defined as the area under the right tail of distribution, if the rejection region in on the right tail; if the rejection region is on the left tail, then the p-value is the area under the left tail (in one-sided alternative hypotheses).

2. P-value for Two-sided Alternative Hypotheses: If the alternative hypothesis is two-sided (that is, rejection regions are both on the left and on the right tails), then the p-value is the area under the right tail or to the left tail of the distribution, depending on whether the computed statistic is closer to the right rejection region or left rejection region.

   For symmetric densities (such as t-density), the left and right tails p-values are the same. However, for non-symmetric densities (such as Chi-square) use the smaller of the two. This makes the test more conservative. Notice that, for a two sided-test alternative hypotheses, the p-value is never greater than 0.5.

3. After finding the p-value as defined here, you compare it with a pre-set a value for one-sided tests, and with a/2 for two sided-test. The larger the p-value is when compared with a (in one-sided alternative hypothesis, and a/2 for the two sided alternative hypotheses), the less evidence we have for rejecting the null hypothesis.

To avoid looking-up the p-values from the limited statistical tables given in your textbook, most professional statistical packages such as SAS and SPSS provide the two-tailed p-value. Based on where the rejection region is, you must find out what p-value to use.

Some textbooks have many misleading statements about p-value and its applications. For example, in many textbooks you find the authors double the p-value to compare it with a when dealing with the two-sided test of hypotheses. One wonders how they do it in the case when "their" p-value exceeds 0.5? Notice that, while it is correct to compare the p-value with a for a one sided tests of hypotheses a, for two-sided hypotheses, one must compare the p-value with a/2, NOT a with 2 times p-value, as some textbooks advise. While the decision is the same, there is a clear distinction here and an important difference, which the careful reader will note.

How to set the appropriate a value? You may have wondered why a = 0.05 is so popular in a test of hypothesis. a = 0.05 is traditional for tests, but is arbitrary in its origins suggested by R.A. Fisher, who suggested it in the spirit of 0.05 being the biggest p-value at which one would think maybe the null hypothesis in a statistical experiment was to be considered false. This was also a tradeoff between "type I error" and "type II error"; that we do not want to accept the wrong null hypothesis, but we do not want to fail to reject the false null hypothesis, either. As a final note, the average of these two p-values is often called the mid-p value.

Conversions from two-sided to one-sided probabilities: Let C be the probability for a two-sided confidence interval (CI) constructed for an estimate. The probability (C₁) that either the estimate is greater than the lower limit or that it is less than the upper limit can be computed by using:

C₁ = C/2 + 1/2,       for conversion to one-sided

Numerical Example: Suppose you wish to convert a C = 90% two-sided CI into a one-sided, then C₁ = 0.90/2 + 1/2 = 95%.

You might need to use Sample Size Determination JavaScript at the design stage of your statistical investigation in decision making with specific, subjective requirements.

---

Bonferroni Method for Multiple P-Values Procedure

One may combine several t-tests by using the Bonferroni method. It works reasonably well when there are only a few tests, but as the number of comparisons increases above 8, the value of 't' required to conclude that a difference exists becomes much larger than it really needs to be, and the method becomes over conservative.

One way to make the Bonferroni t-test less conservative is to use the estimate of the population variance computed from within the groups in the analysis of variance.

t = ( 1 -2 )/ ( s² / n1 + s² / n2 )^1/2,

where s² is the population variance computed within the groups.

Hommel's Multiple P-Values Procedure: This test can be summarized as follows:

Suppose we have n number of P-values: p(i), i =1, .., n, in ascending order corresponding to independent tests. Let j be the largest integer, such as:

p(n-j+k) > ka/j,    for all k=1,..,j.

If no such j exists, reject all hypotheses; otherwise, reject all hypotheses with p(i) £ a / j. This provides a strong control of the family-wise error rate at a level.

There are other improvements on the Bonferroni adjustment when multiple tests are independent or positively dependent. However, the Hommel's method is the most powerful compared with other methods.

Further Readings:
Hommel G., Bonferroni procedures for logically related hypotheses, Journal of Statistical Planning and Inference, 82, 119-128, 1999.
Kost J., and M. McDermott, Combining dependent P-values, Statistics and Probability Letters, 60, 183-190, 2002.
Wasteful P., and S. Young, Resembling-Based Multiple Testing: Examples and Methods for P-Value Adjustment, Wiley, 1992.
Wright S., Adjusted P-values for simultaneous inference, Biometrics, 48, 1005-1013, 1992.

---

Power of a Test and the Size Effect

The power of a test plays the same role in hypothesis testing that Standard Error played in estimation. It is a measuring tool for assessing the accuracy of a test or in comparing two competing test procedures.

The power of a test is the probability of rejecting a false null hypothesis when the null hypothesis is false. This probability is inversely related to the probability of making a Type II error, not rejecting the null hypothesis when it is false. Recall that we choose the probability of making a Type I error when we set a. If we decrease the probability of making a Type I error, then we increase the probability of making a Type II error. Therefore, there are basically two errors possible when conducting a statistical analysis; type I error and and type II error:

• Type I error - (producer's) risk of rejecting the null hypothesis when it is in fact true.
• Type II error - (consumer's) risk of not rejecting the null hypothesis when it is in fact false.

Power and Alpha (a): Thus, the probability of not rejecting a true null has the same relationship to Type I errors as the probability of correctly rejecting an untrue null does to Type II error. Yet, as I mentioned if we decrease the odds of making one type of error we increase the odds of making the other type of error. What is the relationship between Type I and Type II errors? For a fixed sample size, decreasing one type of error increases the size of the other one.

Power and the Size Effect: Anytime we test whether a sample differs from a population, or whether two samples come from 2 separate populations, there is the condition that each of the populations we are comparing has its own mean and standard deviation (even if we do not know it). The distance between the two population means will affect the power of our test. This is known as the size of treatment, also known as the effect size, as shown in the following table with the three popular values for a:

Power as a Function of a and the Size Effect
	a
Size Effect 0.10 0.05 0.01       1.0 .22  .13  .03    2.0 .39  .26  .09    3.0 .59  .44  .20    4.0 .76  .64  .37    5.0 .89  .79  .57    6.0 .96  .91  .75    7.0 .99  .97  .88

Power and the Size of Variance s²: The greater the variance S², the lower the power 1-b. Anything that effects the extent to which the two distributions share common values will increase b (the likelihood of making a Type II error)

Power and the Sample Size: The smaller the sample sizes n, the lower the power. Very small n produces power so low that false hypotheses are accepted.

The following is a list of four factors influencing the power:

• effect size (for example, the difference between the means)
• variance S²
• significance level a
• number of observations, or the sample size n

In practice, the first three factors are often fixed. Only the sample size can be controlled by the statistician and that only within budget constraint. There exists a tradeoff between budget and achievement of desirable accuracy in any analysis.

A Numerical Example: The power of a test is most easily understood by viewing it in the context of a composite test. A composite test requires the specification of a population mean as the alternative hypothesis. For example, using Z-test of hypothesis in the following Figure. The power is developed from specification of an alternative hypothesis such as m = 2.5, and m = 3. The resultant distribution under this alternative shifts to the right 2.5 units with the shaded area representing the power of the test, correctly rejecting a false null.

Power of a Test
Click on the image to enlarge it

Not rejecting the null hypothesis when it is false is defined as a Type II error, and is denoted by the b region. In the above Figure this region lies to the left of the critical value. In the configuration shown in this Figure, b falls to the left of the critical value (and below the statistic's density (or probability) function under the alternative hypothesis H_a). The b is also defined as the probability of not-rejecting a false null hypothesis when it is false, also called a miss. Related to the value of b is the power of a test. The power is defined as the probability of rejecting the null hypothesis given that a specific alternative is true, and is computed as (1- a).

A Short Discussion: Consider testing a simple null versus simple alternative. In the Neyman-Pearson setup, an upper bound is set for the probability of a given Type I error (a), and then it is desirable to find tests with low probability of type II error (b) given this. The usual justification for this is that "we are more concerned about a Type I error, so we set an upper limit on the a that we can tolerate." I have seen this sort of reasoning in elementary texts and also in some advanced ones. It doesn't seem to make any sense. When the sample size is large, for most standard tests, the ratio b/a tends to 0. If we care more about Type I error than Type II error, why should this concern dissipate with increasing sample size?

This is indeed a drawback of the classical theory of testing statistical hypotheses. A second drawback is that the choice lies between only two test decisions: reject the null or accept the null. It is worth considering approaches that overcome these deficiencies. This can be done, for example, by the concept of profile-tests at a 'level' a. Neither the Type I nor Type II error rates are considered separately, but they are the ratio of a correct decision. For example, we accept the alternative hypothesis H_a and reject the null H₀, if an event is observed which is at least a-times greater under H_a than under H₀. Conversely, we accept H₀ and reject H_a, if an event is observed which is at least a-times greater under H₀ than under H_a. This is a symmetric concept which is formulated within the classical approach.

Power of Parametric versus Non-parametric Tests: As a general rule, for a given sample size n, the parametric tests are more powerful than their non-parametric counterparts. The primarily reason for this is that we have emphasized parametric tests. Moreover, among the parametric tests, those which use correlation are more powerful, such as the before-and-after test. This is known as a Variance Reduction Technique used in system simulation to increase the accuracy (i.e., reduce variation) without increasing the sample size.

Correlation Coefficient as a Measuring Tool and Decision Criterion for the Effect Size: The correlation coefficient could be obtained and used as a measuring tool and decision criteron for the strength of the effect size based on the computed test-statistic for major hypothesis testing.

The correlation coefficient r stands as a very useful and accessible index of the magnitude of effect. It is commonly accepted that the small, medium, and large effect sizes correspond to r-values over 0.1, 0.3, and 0.5, respectively. The following are needed transformation of some major inferential statistics to the r-value:

• For the t(df)-statistic:            r = [t²/(t² + df)]^½

• For the F(1,df₂)-statistic:         r = [F/(F + df)]^½

• For the c²(1)-statistic:            r = [c²/n] ^½

• For the Standard Normal Z:     r = (Z²/n)^½

You might like to use Sample Size Determination JavaScript at the design stage of your statistical investigation in decision making with specific subjective requirements.

Further Reading:
Murphy K., and B. Myors, Statistical Power Analysis, L. Erlbaum Associates, 1998.

---

Parametric vs. Non-Parametric vs. Distribution-free Tests

One must use a statistical technique called non-parametric if it satisfies at least one of the following five types of criteria:

1. The data entering the analysis are enumerative; that is, counted data represent the number of observations in each category or cross-category.

2. The data are measured and/or analyzed using a nominal scale of measurement.

3. The data are measured and/or analyzed using an ordinal scale of measurement.

4. The inference does not concern a parameter in the population distribution; for example, the hypothesis that a time-ordered set of observations exhibits a random pattern.

5. The probability distribution of the statistic upon which the analysis is based is not dependent upon specific information or conditions (i.e., assumptions) about the population(s) from which the sample(s) are drawn, but only upon general assumptions, such as a continuous and/or symmetric population distribution.

According to these creteria, the distinction of non-parametric is accorded either because of the level of measurement used or required for the analysis, as in types 1 through 3; the type of inference, as in type 4, or the generality of the assumptions made about the population distribution, as in type 5.

For example, one may use the Mann-Whitney Rank Test as a non-parametric alternative to Students T-test when one does not have normally distributed data.

Mann-Whitney: To be used with two independent groups (analogous to the independent groups t-test)
Wilcoxon: To be used with two related (i.e., matched or repeated) groups (analogous to the related samples t-test)
Kruskall-Wallis: To be used with two or more independent groups (analogous to the single-factor between-subjects ANOVA)
Friedman: To be used with two or more related groups (analogous to the single-factor within-subjects ANOVA)

Non-parametric vs. Distribution-free Tests:

Non-parametric tests are those used when some specific conditions for the ordinary tests are violated.

Distribution-free tests are those for which the procedure is valid for all different shape of the population distribution.

For example, the Chi-square test concerning the variance of a given population is parametric since this test requires that the population distribution be normal. The Chi-square test of independence does not assume normality condition, or even that the data are numerical. The Kolmogorov-Smirnov test is a distribution-free test, which is applicable to comparing two populations with any distribution of continuous random variable.

The following section is an interesting non-parametric procedure with various and useful applications.

Comparison of Two Random Variables: Consider two independent observations X = (x₁, x₂,…, x_r) and Y = (y₁, y₂,…, y_s) for two random variables X and Y respectively. To estimate the reliability function:

R = Pr (X > Y)

One may use:

The estimator RS = U/(r ´ s),

where U is the number of pairs (x_i, y_j) such that x_i > y_j, for all i = 1, 2, ,r,  and j = 1, 2,..,s.

This estimator is an unbiased one with the minimum variance for R. It is important to know that the estimate has an upper limit, non-negative delta value for its accuracy:

Pr{R ³ RS - d} ³ max {1- exp(-2nd²), 4nd²/(1-4nd²)}.

Application areas include the insurance ruin problem. Let random variable Y denote the claims per unit of time and let random variable X denote the return on investment (ROI) for the Insurance Company. Finally, let z denote the constant premium amount collected; then the probability that the insurance company will survive is:

R = Pr [X + z > Y}.

You might like to use the Kolmogorov-Smirnov Test for Two Populations and Comparing Two Random Variables in checking your computations and performing some numerical experiment for a deeper understanding of these concepts.

Further Readings:
Arsham H., A generalized confidence region for stress-strength reliability, IEEE Transactions on Reliability, 35(4), 586-589, 1986.
Conover W., Practical Nonparametric Statistics, Wiley, 1998.
Hollander M., and D. Wolfe, Nonparametric Statistical Methods, Wiley, 1999.
Kotz S., Y. Lumelskii, and M. Pensky, The Stress-Strength Model and Its Generalizations: Theory and Applications,
Imperial College Press, London, UK, 2003, distributed by World Scientific Publishing.

---

Hypotheses Testing

Remember that, in the t-tests for differences in means, there is a condition of equal population variances that must be examined. One way to test for possible differences in variances is to do an F test. However, the F test is very sensitive to violations of the normality condition; i.e., if populations appear not to be normal, then the F test will tend to reject too often the null of no differences in population variances.

You might like to use the following JavaScript to check your computations and to perform some statistical experiments for deeper understanding of these concepts:

• Testing the Mean.
• Testing the Variance.
• Testing Two Populations.
• Testing the Difference: The Before-and-After Test.
• ANOVA.
• For statistical equality of two populations, you might like to use the Kolmogorov-Smirnov Test.

---

Single Population t-Test

The purpose is to compare the sample mean with the given population mean. The aim is to judge the claimed mean value, based on a set of random observations of size n. A necessary condition for validity of the result is that the population distribution is normal, if the sample size n is small (say less than 30).

The task is to decide whether to accept a null hypothesis:

H₀ = m = m₀

or to reject the null hypothesis in favor of the alternative hypothesis:

H_a: m is significantly different from m₀

The testing framework consists of computing a the t-statistics:

T = [( - m₀) n^1/2] / S

Where is the estimated mean and S² is the estimated variance based on n random observations.

The above statistic is distributed as a t-distribution with parameter d.f. = n = (n-1). If the absolute value of the computed T-statistic is "too large" compared with the critical value of the t-table, then one rejects the claimed value for the population's mean.

This test could also be used for testing similar claims for other unimodal populations including those with discrete random variables, such as proportion, provided there are sufficient observations (say, over 30).

You might like to use Testing the Mean JavaScript in checking your computations. and Sample Size Determination JavaScript at the design stage of your statistical investigation in decision making with specific subjective requirements.

You might like also to use JavaScript Testing Two Populations.

---

Pooling the Sampling Estimates for Mean, Variance, and Standard Deviation

If an estimate is an unbiased such as sample mean, then it is a good idea to pool the estimates to get a single estimate from several relatively small samples. The pooled estimate is a “good” estimate when compared with each individual estimates.

Pooled Mean: Supposed we have m number of estimates (i), of sample size n(i), for the population expected value m, the pooled estimate is:

[S n(i)(i)] / [Sn(i)], both sums are over all values of i = 1, 2,. . ., m.

Pooled Variance: Since the sample variance is also unbiased estimate of population variance s², therefore, it is a good idea to pool the estimates to get a single estimate from m number of estimates S(i)², of sample size n(i), the pooled estimate is:

{[S [n(i) – 1] S(i)² ] } / {[S n(i)] – m}, both sums are over all values of i = 1, 2,…, m.

We pool variance estimates for other good reasons. Depending on a particular reason, then the conclusion might have to be made explicitly conditional on e.g., the validity of the equal-variance model. There are several different good reasons for pooling:

• to get a single stable estimate from several relatively small samples, where variance fluctuations seem not to be systematic; or

• for convenience, when all the variance estimates are near enough to equality; or

• when there is no choice but to model variance, as in simple linear regression with no replicated X values.

You might like to use JavaScript Pooling the Means, and Variances.

Pooled Standard Deviation: Both the sample mean, and variance are unbiased estimates for the population parameters, m, and s², respectively, however the sample standard deviation in NOT an unbiased estimate of population standard deviation s. This is so, because of an equality known as the Jensen's inequality when applied to a concave function, i.e., the square root of the unbiased variance estimate. Therefore, pooling standard deviation directly is meaningless; the best one can do to take the square root of the pooled variance

Notice that, when sample sizes are large and nearly equal, so that there is essentially no difference between the pooled and unpooled estimates of standard errors of paired-data samples, and degrees of freedom are nearly asymptotic. This rationale can fall apart for any other cases. One must pool variance rather than merely taking a shortcut in the computation of standard errors.

If you calculate the test without the assumption, you have to determine the degrees of freedom (d.f.). The formula works in such a way that d.f. will be less if the larger sample variance is in the group with the smaller number of observations. This is the case in which the two tests will differ considerably. A study of the formula for the d.f. is most enlightening, and one must understand the correspondence between the unfortunate design, having the most observations in the group with little variance, and the low d.f. and accompanying large t-value.

Applications: When doing t tests for differences in means of populations, for independent samples case:

1. For differences in means that do not make any assumption about equality of population variances, use the standard error formula:
   [S²₁/n₁ + S²₂/n₂]^½,

   with d.f. = n = n₁ or n₂ whichever is smaller.

2. With equal variances, use the statistics:
   

   with parameter d.f. = n = (n₁ + n₂- 2), for n₁, and n₂ greater than to 1, where the pooled variance is:

   

3. If total N is less than 50 and one sample is 1/2 the size of the other (or less), and if the smaller sample has a standard deviation at least twice as large as the other sample, then apply the procedure given in item no. 1, but adjust d.f. parameter of the t-test to the largest integer less than or equal to:
   d.f. = n = A/(B +C),

   where:

   A = [S²₁/n₁ + S²₂/n₂]²,

   B = [S²₁/n₁]² / (n₁ -1),

   C = [S²₂/n₂]²/ (n₂ -1)

   Otherwise, do not worry about the problem of having an actual a level that is much different than what you have set it to be.

Statistics with Confidence Section is concerned with the construction of a confidence interval where the equality of variances condition is an important issue.

The last approach, which is very general with conservative results, can be implemented using Testing Two Populations JavaScript.

You might like to use JavaScript Testing the Difference in Means: The Before-and-After Test and Paired Proportion Test for dependent proportions.

---

Non-parametric Multiple Comparison Procedures

Duncan's multiple-range test: This is one of the many multiple comparison procedures. It is based on the standardized range statistic by comparing all pairs of means while controlling the overall Type I error at a desirable level. While it does not provide interval estimates of the difference between each pair of means, it does indicate which means are significantly different from the others. For determining the significant differences between a single control group mean and the other means, one may use the Dunnett's multiple-comparison test.

---

Introduction to Tests for Statistical Equality of Two or More Populations:

Two random variables X and Y having distribution F_X(x) and F_Y(y) respectively, are said to be equivalent, or equal in law, or equal in distribution, if and only if they have the same distribution function. That is,

F_X(z) = F_Y(z),   for all z,

There are different tests depending on the intended applications. The widely used tests for statistical equality of populations are as follow:

1. Equality of Two Normal Populations: One may use the Z-test and F-test to check the equality of the means, and the equality of variances, respectively.

2. Testing a Shift in Normal Populations: Often we are interested in testing for a given shift in a given population distribution, that is testing if a random variable Y is equal in distribution to another X + c for some constant c. In other words, the distribution of Y is the distribution of X shifted. In testing any shift in distribution one needs to test for normality first, and then testing the difference in expected values by applying the two-sided Z-test with the null hypothesis of:
   H₀:   m_Y - m_X = c.

3. Analysis of Variance: Analysis of Variance (ANOVA) tests are designed for simultaneous testing of equality of three or more populations. The preconditions in applying ANOVA are normality of each population's distribution, and the equality of all variances simultaneously (not the pair-wise tests).

   Notice that ANOVA is an extension of item no. 1 in testing equality of more than two populations. It can be shown that if one applies ANOVA for testing the equality of two populations based on two independent samples with sizes of n1 and n2 form each population, respectively, then the results of both tests will be identical. Moreover, the test-statistic obtained by each test are directly related, i.e.,

   F _{a , (1, n1+ n2 - 2)}  =   t ² _{a/2 , (n1+ n2 - 2)}

4. Equality of Proportions in Several Populations: This test is for discrete random variables. It is one of the many interesting chi-square applications.

5. Distribution-free Equality of Two Populations: Whenever one is interested in testing the equality of two populations with a common continuous random variable, without any reference to the underlying distribution such as normality condition, one may use the distribution-free known as the K-S test.

6. Non-parametric Comparison of Two Random Variables: Consider two independent observations X = (x₁, x₂,…, x_r) and Y = (y₁, y₂,…, y_s) for two independent populations with random variables X and Y, respectively. Often we are interested in estimating the Pr (X > Y).

---

Equality of Two Normal Populations:

The normal or Gaussian distribution is a continuous symmetric distribution that follows the familiar bell-shaped curve. One of its nice features is that, the mean and variance uniquely and independently determines the distribution.

Therefore, for testing the statistical equality of two independent normal populations, one must first perform the Lilliefors' Test for Normality to assess this condition. Given that both populations are normally distributed, then one must performing two more tests, namely the test for equality of the two means and the test for equality of the two variances. Both of these tests can be carried out by using the Test of Hypotheses for Two Populations JavaScript.

---

Multi-Means Comparisons: Analysis of Variance (ANOVA)

The tests we have learned up to this point allow us to test hypotheses that examine the difference between only two means. Analysis of Variance or ANOVA will allow us to test the difference between two or more means. ANOVA does this by examining the ratio of variability between two conditions and variability within each condition. For example, say we give a drug that we believe will improve memory to a group of people and give a placebo to another group of people. We might measure memory performance by the number of words recalled from a list we ask everyone to memorize. A t-test would compare the likelihood of observing the difference in the mean number of words recalled for each group. An ANOVA test, on the other hand, would compare the variability that we observe between the two conditions to the variability observed within each condition. Recall that we measure variability as the sum of the difference of each score from the mean. When we actually calculate an ANOVA we will use a short-cut formula

Thus, when the variability that we predict between the two groups is much greater than the variability we don't predict within each group, then we will conclude that our treatments produce different results.

An Illustrative Numerical Example for ANOVA

Consider the following (small integers, indeed for illustration while saving space) random samples from three different populations.

With the null hypothesis:
H₀: µ1 = µ2 = µ3,
and the alternative:
H_a: at least two of the means are not equal.

At the significance level a = 0.05, the critical value from F-table is
F _{0.05, 2, 12} = 3.89.

Sum Mean Sample P1 2 3 1 3 1 10 2 Sample P2 3 4 3 5 0 15 3 Sample P3 5 5 5 3 2 20 4

Demonstrate that, SST=SSB+SSW.
That is, the sum of squares total (SST) equals sum of squares between (SSB) the groups plus sum of squares within (SSW) the groups.

Computation of sample SST: With the grand mean = 3, first, start with taking the difference between each observation and the grand mean, and then square it for each data point.

Sum Sample P1 1 0 4 0 4 9 Sample P2 0 1 0 4 9 14 Sample P3 4 4 4 0 1 13

Therefore SST = 36 with d.f = (n-1) = 15-1 = 14

Computation of sample SSB:

Second, let all the data in each sample have the same value as the mean in that sample. This removes any variation WITHIN. Compute SS differences from the grand mean.

Sum Sample P1 1 1 1 1 1 5 Sample P2 0 0 0 0 0 0 Sample P3 1 1 1 1 1 5

Therefore SSB = 10, with d.f = (m-1)= 3-1 = 2 for m=3 groups.

Computation of sample SSW:

Third, compute the SS difference within each sample using their own sample means. This provides SS deviation WITHIN all samples.

Sum Sample P1 0 1 1 1 1 4 Sample P2 0 1 0 4 9 14 Sample P3 1 1 1 1 4 8

SSW = 26 with d.f = 3(5-1) = 12. That is, 3 groups times (5 observations in each -1)

Results are: SST = SSB + SSW, and d.f_SST = d.f_SSB + d.f_SSW, as expected.

Now, construct the ANOVA table for this numerical example by plugging the results of your computation in the ANOVA Table. Note that, the Mean Squares are the Sum of squares divided by their Degrees of Freedom. F-statistics is the ratio of the two Mean Squares.

The ANOVA Table
Sources of Variation	Sum of Squares	Degrees of Freedom	Mean Squares	F-Statistic
Between Samples	10	2	5	2.30
Within Samples	26	12	2.17
Total	36	14

Conclusion: There is not enough evidence to reject the null hypothesis H₀.

The Logic behind ANOVA: First, let us try to explain the logic and then illustrate it with a simple example. In performing the ANOVA test, we are trying to determine if a certain number of population means are equal. To do that, we measure the difference of the sample means and compare that to the variability within the sample observations. That is why the test statistic is the ratio of the between-sample variation (MSB) and the within-sample variation (MSW). If this ratio is close to 1, there is evidence that the population means are equal.

Here is a good application for you: Many people believe that men get paid more in the business world than women, simply because they are male. To justify or reject such a claim, you could look at the variation within each group (one group being women's salaries and the other group being men's salaries) and compare that to the variation between the means of randomly selected samples of each population. If the variation in the women's salaries is much larger than the variation between the men's and women's mean salaries, one could say that because the variation is so large within the women's group that this may not be a gender-related problem.

Now, getting back to our numerical example of the drug treatment to improve memory vs the placebo. We notice that: given the test conclusion and the ANOVA test's conditions, we may conclude that these three populations are in fact the same population. Therefore, the ANOVA technique could be used as a measuring tool and statistical routine for quality control as described below using our numerical example.

Construction of the Control Chart for the Sample Means: Under the null hypothesis, the ANOVA concludes that µ1 = µ2 = µ3; that is, we have a "hypothetical parent population." The question is, what is its variance? The estimated variance (i.e., the total mean squares) is 36 / 14 = 2.75. Thus, estimated standard deviation is = 1.60 and estimated standard deviation for the means is 1.6 / 5^½ = 0.71. Under the conditions of ANOVA, we can construct a control chart with the warning limits = 3 ± 2(0.71); the action limits = 3 ± 3(0.71). The following figure depicts the control chart.

You might like to use ANOVA: Testing Equality of Means for your computations, and then to interpret the results in managerial (not technical) terms.

You might need to use Sample Size Determination JavaScript at the design stage of your statistical investigation in decision making with specific subjective requirements.

---

ANOVA for Normal but Condensed Data Sets

In testing the equality of several means, often the raw data are not available. In such a case, one must perform the needed analysis based on secondary data using the data summaries; namely, the triple-set: The sample sizes, the sample means, and the sample variances.

Suppose one of the samples is of size n having the sample mean , and the sample variance S². Let:

y_i = + (S²/n)^½     for all i = 1, 2, …, n-1,

and

y_n = n - (n - 1)y₁

Then, the new random data y_i's are surrogate data having the same mean and variance as the original data set. Therefore, by generating the surrogate data for each sample, one can perform the standard ANOVA test. The results are identical.

You might like to use ANOVA for Condensed Data for your computation and experimentation.

The JavaScript Subjective Assessment of Estimates tests the claim that at least the ratio of one estimate to the largest estimate is as large as a given claimed value.

Further Reading:
Larson D., Analysis of variance with just summary statistics as input, The American Statistician, 46(2), 151-152, 1992.

---

ANOVA for Dependent Populations

Populations can be dependent in either of the following ways:

1. Every subject is tested in every experimental condition. This kind of dependency is called the repeated-measurement design.
2. Subjects under different experimental conditions are related in some manner. This kind of dependency is called matched-subject designed.

An Application: Suppose we are interested in studying the effect of alcohol on driving ability. Ten subjects are given three different alcohol levels and the number of driving errors are tabulated below:

											Mean
0 oz	2	3	1	3	1	4	1	3	2	1	2.1
2 oz	3	2	1	4	2	3	1	5	1	2	2.4
4 oz	3	1	2	4	2	5	2	4	3	2	3.1

The test null hypothesis is:

H₀: µ1 = µ2 = µ3,

and the alternative:

H_a: at least two of the means are not equal.

Using the ANOVA for Dependent Populations JavaScripts, we obtain the needed information in constructing the following ANOVA table:

The ANOVA Table
Sources of Variation	Sum of Squares	Degrees of Freedom	Mean Squares	F-Statistic
Subjects	31.50	9	3.50	-
Between	5.26	2	2.63	7.03
Within	6.70	18	0.37
Total	43.46	29

Conclusion: The p-value is P= 0.006, indicating a strong evidence against the null hypothesis. The means of the populations are not equal. Here, one may conclude that person who has consumed more than certain level of alcohol commits more driving errors.

A "block design sampling" implies studying more than two dependent populations. For testing the equality of means of more than two populations based on block design sampling, you may use Two-Way ANOVA Test JavaScript. In the case of having block design data with replications, use Two-Way ANOVA with Replications JavaScript to obtain the needed information for constructing the ANOVA tables.

---

Test for Equality of Several Population Proportions

The Chi-square test of homogeneity provides an alternative method for testing the null hypothesis that two population proportions are equal. Moreover, it extend, to several populations similar to the ANOVA test that compares several means.

An Application: Suppose we wish to test the null hypothesis

H₀: P₁ = P₂ = ..... = P_k

That is, all three population proportions are almost identical. The sample data from each of the three populations are given in the following table:

Test for homogeneity of Several Population Proportions
Populations	Yes	No	Total
Sample I	60	40	100
Sample II	57	53	110
Sample III	48	72	120
Total	165	165	330

The Chi-square statistic is 8.95 with d.f. = (3-1)(3-1) = 4. The p-value is equal to 0.062, indicating that there is moderate evidence against the null hypothesis that the three populations are statistically identical.

You might like to use Testing Proportions to perform this test.

---

Distribution-free Equality of Two Populations

For statistical equality of two populations, one may use the Kolmogorov-Smirnov Test (K-S Test) for two populations. The K-S test seeks differences between the two population's distribution function based on their two independent random samples. The test rejects the null hypothesis of no difference between the two populations if the difference between the two empirical distribution functions is "large".

Prior to applying the K-S test it is necessary to arrange each of the two sample observations in a frequency table. The frequency table must have a common classification. Therefore the test is based on the frequency table, which belongs to the family of distribution-free tests.

The K-STest process is as follows:

1. Some k number of "classes" is selected, each typically covering a different but similar range of values.

2. Some much larger number of independent observations (n₁, and n₂, both larger than 40) are taken. Each is measured and its frequency is recorded in a class.

3. Based on the frequency table, the empirical cumulative distribution functions F1_i and F2_i for two sample populations are constructed, for i = 1, 2,..., k.

4. The K-S statistic is the largest absolute difference between F1_i and F2_i; i.e.,

   K-S statistic = D = Maximum | F1_i - F2_i |,        for all i = 1, 2, .., k.

The critical values of K-S statistic can be found at Computers and Computational Statistics with Applications

An Application: The daily sales of the two subsidiaries of The PC & Accessories Company are shown in the following table, with n1 = 44, and n2 = 54:

Daily Sales at Two Branches Over 6 Months
Sales ($1000)		Frequency I	Frequency II
0 - 2		11	1
3 - 5		7	3
6 - 8		8	6
9 - 11		3	12
12 - 14		5	12
15 - 17		5	14
18 - 20		5	6
Sums		44	54

The manager of the first branch is claiming that "since the daily sales are random phenomena, my overall performance is as good as the other manager's performance." In other words:

H₀: The daily sales at the two stores are almost the same.
H_a: The performance of the managers is significantly different.

Following the above process for this test, the K-S statistic is 0.421 with the p-value of 0.0009, indicating a strong evidence against the null hypothesis. There is enough evidence that the performance of the manager of the second branch is better.

---

Introduction to Applications of the Chi-square Statistic

The variance is not the only thing for which you use a Chi-square test for.

The most widely used applications of Chi-square distribution are:

The Chi-square Test for Association which is a non-parametric test; therefore, it can be used for nominal data too. It is a test of statistical significance widely used bivariate tabular association analysis. Typically, the hypothesis is whether or not two populations are different in some characteristic or aspect of their behavior based on two random samples. This test procedure is also known as the Pearson Chi-square test.

The Chi-square Goodness-of-Fit Test is used to test if an observed distribution conforms to any particular distribution. Calculation of this goodness-of-fit test is by comparison of observed data with data expected based on a particular distribution.

One of the disadvantages of some of the Chi-square tests is that they do not permit the calculation of confidence intervals; therefore, determination of the sample size is not readily available.

Treatment of Cases with Many Categories: Notice that, although in the following section most of the crosstables have only two categories, it is always possible to convert cases with many categories into similar crosstables. To do so, one must consider all possible pairs of categories and their numerical values while constructing the equivalent "two-categories" crosstable.

---

Test for Crosstable Relationship

Crosstables: Often crosstables are used to test relationships among two categorical types of data, or independence of two variables, such as cigarette smoking and drug use. If you were to survey 1000 people on whether or not they smoke and whether or not they use drugs, you would get one of four answers: (no, no) (no, yes) (yes, no) (yes, yes)

By compiling the number of people in each category, you can ultimately test whether drug usage is independent of cigarette smoking by using the Chi-square distribution (this is approximate, but works well). Again, the methodology for this is in your textbook. The degrees of freedom is equal to (number of rows-1)(number of columns -1). That is, these many numbers needed to fill in the entire body of the crosstable, the rest will be determined by using the given row sums and the column sums values.

Do not forget the conditions for the validity of Chi-square test and related expected values greater than 5 in 80% or more of the cells. Otherwise, one could use an "exact" test, using either a permutation or resampling approach.

Using Chi-square in a 2x2 table requires the Yates's correction. One first subtracts 0.5 from the absolute differences between observed and expected frequencies for each of the three genotypes before squaring, dividing by the expected frequency, and summing. The formula for the Chi-square value in a 2x2 table can be derived from the Normal Theory comparison of the two proportions in the table using the total incidence to produce the standard errors. The rationale of the correction is a better equivalence of the area under the normal curve and the probabilities obtained from the discrete frequencies. In other words, the simplest correction is to move the cut-off point for the continuous distribution from the observed value of the discrete distribution to midway between that and the next value in the direction of the null hypothesis expectation. Therefore, the correction essentially only applied to one d.f. tests where the "square root" of the Chi-square looks like a "normal/t-test" and where a direction can be attached to the 0.5 addition.

Chi-square distribution is used as an approximation of the binomial distribution. By applying a continuity correction, we get a better approximation of the binomial distribution for the purposes of calculating tail probabilities.

Given the following 2x2 table, one may compute some relative risk measures:

a	b
c	d

The most usual measures are:

Rate-difference: a/(a+c) - b/(b+d)
Rate-ratio: (a/(a+c))/(b/(b+d))
Odds-ratio: ad/bc

The rate difference and rate ratio are appropriate when you are contrasting two groups whose sizes (a+c and b+d) are given. The odds ratio is for when the issue is association rather than difference.

The risk-ratio (RR) is the ratio of the proportion (a/(a+b)) to the proportion (c/(c+d)):

RR = (a / (a + b)) / (c / (c + d))

RR is thus a measure of how much larger the proportion in the first row is compared to the second. RR value of < 1.00 indicating a 'negative' association [a/(a+b) < c/(c+d)], 1.00 indicating no association [a/(a+b) = c/(c+d)], and >1.00 indicating a 'positive' association [a/(a+b) > c/(c+d)]. The further from 1.00 the RR is, the stronger the association.

An Application: Suppose a counselor of a school in a small town is interested whether the curriculum chosen by students is related to the occupation of their parents. It is necessary to record the data as shown in the following contingency table with two rows (r1, r2) and three columns (c1, c2, c3):

Relationship between occupation of parents and curriculum chosen by high school students

Curriculum Chosen by Students

Parental Occupation College prep Vocational General Totals Professional 12 2 6 6 6 8 20 Blue collar 20 Totals 18 8 14

Under the hypothesis that there is no relation, the expected (E) frequency would be:

E_{i, j} = (Sr_i)(Sc_j)/N

The Observed (O) and Expected (E) frequencies are recorded in the following table:

Expected frequencies for the data.

College prep Vocational General Totals Professional O = 12 E = 9 O = 2 E = 4 O = 6 E = 7 O = 6 E = 9 O = 6 E = 4 O = 8 E = 7 åO = 20 åE = 20 Blue collar åO= 20 åE = 20 Totals åO = 18 åE = 18 åO = 8 åE = 8 åO = 14 åE = 14

The quantity

c ² = S [(O - E )² / E]

is a measure of the degree of deviation between the Observed and Expected frequencies. If there is no relationship between the row variable and the column variable this measure will be very close to zero. Under the hypothesis that there is a relationship between the rows and the columns, this quantity has a Chi-square distribution with parameter equal to number of rows minus 1, multiplied by number of columns minus 1.

For this numerical example we have:

c ² = S [(O - E )² / E] = 30/7 = 4.3

with d.f. = (2-1)(3-1) = 2, that has the p-value of 0.14, suggesting little or no real evidences against the null hypothesis.

The main question is how large is this measure. The maximum value of this measure is:

c ²_max = N(A-1),

where A is the number of rows or columns, whichever is smaller. For our numerical example it is, 40(2-1) = 40.

The coefficient of determination which has a range of [0, 1], provides relative strength of relationship, computed as

c ²/c ²_max = 4.3/40 = 0.11

Therefore we conclude that the degree of association is only 11% which is fairly weak.

Alternatively, you could also look at the contingency coefficient f statistic, which is:

f = [ c²/(N + c²)]^½ = 0.31

This statistic ranges between 0 and 1 and can be interpreted like the correlation coefficient. This measure also indicates that the curriculum chosen by students is related to the occupation of their parents.

You might like to use Chi-square Test for Crosstable Relationship in performing this test, and he P-values for the Popular Distributions JavaScript to findout the p-values of Chi-square statistic.

Further Readings:
Agresti A., Categorical Data Analysis, Wiley, 2002.
Fleiss J., Statistical Methods for Rates and Proportions, Wiley, 1981.

---

Identical Populations Test for Crosstable Data

Test of homogeneity is much like the Test for Crosstable Relationship in that both deal with the cross-classification of nominal data; that is, r ´ c tables. The method of computing Chi-square statistic is the same for both tests, with the same d.f.

The two tests differ, however, in the following respect. The Test for Crosstable Relationship is made on data drawn from a single population (with fixed total) where one is concerned with whether one set of attributes is independent of another set. The test for homogeneity, on the other hand, is designed to test the null hypothesis that two or more random samples are drawn from the same population or from different populations, according to some criterion of classification applied to the samples.

The homogeneity test is concerned with the question: Are the samples drawn form populations that are homogeneous (i.e., the same) with respect to some criterion of classification?

In the crosstable for this test, either the row or the column categories may represent the populations from which the samples are drawn.

An Application: Suppose a board of directors of a labor union wishes to survey the opinion of its members regarding a change in its constitution. The following table shows the result of the survey sent to three union locals:

Reactions of A Sample of Three Locals Group Members
Union Local
Reaction A B C In Favor 18 22 10 7 14 9 5 4 11 Against No Response

The problem is not to determine whether or not the union members are in favor of the change. The question is to test if there is a significant difference in the proportions of opinion of the three populations' members concerning the proposed change.

The Chi-square statistic is 9.58 with d.f. = (3-1)(3-1) = 4. The p-value is equal to 0.048, indicating that there is moderate evidence against the null hypothesis that the three union locals are the same.

You might like to use Populations Homogeneity Test to perfor this test.

Further Readings:
Agresti A., Categorical Data Analysis, Wiley, 2002.
Clark Ch., and L. Schkade, Statistical Analysis for Administrative Decisions, South-Western Pub., 1979.

---

Test for Equality of Several Population Medians

Generally, the median provides a better measure of location than the mean when there are some extremely large or small observations; i.e., when the data are skewed to the right or to the left. For this reason, median income is used as the measure of location for the U.S. household income.

Suppose we are interested in testing the equality of the medians of k number of populations with respect to the same continuous random variable.

The first step in calculating the test statistic is to compute the common median of the k samples combined. Then, determine for each group the number of observations falling above and below the common median. The resulting frequencies are arranged in a 2 by k crosstable. If the k samples are, in fact, from populations with the same median, one expects about one half the score in each sample to be above the combined median and about one half to be below. In the case that some observations are equal to the combined median, one may drop those few observations, in constructing a 2 by k crosstable. Under this condition, now the Chi-square statistic may be computed and compared with the p-value of Chi-square distribution with d.f. = k-1.

An illustrative application: Do public and private primary school teachers differ with respect to their salary? The data from a random sample are given in the following table (in thousands of dollars per year).

Public	Private	Public	Private
35	29	25	50
26	50	27	37
27	43	45	34
21	22	46	31
27	42	33
38	47	26
23	42	46
25	32	41

The test of hypothesis is:

H₀: The public and private school teachers' salaries are almost the same.

The median of all data (i.e., combined) is 33.5. Now determine in each group the number of observations falling above and below the common median of 33.5. The resulting frequencies are shown in the following table:

Crosstable for the public and private school teachers'
	Public	Private	Total
Above median	6	8	14
Below median	10	4	14
Total	16	12	28

The Chi-square statistic based on this table is 2.33. The p-value for the computed test statistic with d.f. = (2-1)(2-1) = 1 is 0.127, therefore, we are unable to reject the null hypothesis.

You might like to use Testing Medians to perform this test.

---

Goodness-of-Fit Test for Probability Mass Functions

There are other tests that might use the Chi-square, such as goodness-of-fit test for discrete random variables. Therefore, Chi-square is a statistical test that measures "goodness-of-fit". In other words, it measures how much the observed or actual frequencies differ from the expected or predicted frequencies. Using a Chi-square table will enable you to discover how significant the difference is. A null hypothesis in the context of the Chi-square test is the model that you use to calculate your expected or predicted values. If the value you get from calculating the Chi-square statistic is sufficiently high (as compared to the values in the Chi-square table), it tells you that your null hypothesis is probably wrong.

Let Y₁, Y ₂, . . ., Y _n be a set of independent and identically distributed discrete random variables. Assume that the probability distribution of the Y _i's has the probability mass function f _o (y). We can divide the set of all possible values of Y_i, i = {1, 2, ..., n}, into m non-overlapping intervals D₁, D₂, ...., D_m. Define the probability values p₁, p₂, ..., p_m as;

p₁ = P(Y_i Î D₁)
p₂ = P(Y_i Î D₂)

:

p_m = P(Y_i Î D_m)

Where the symbol Î means, "an element of".

Since the union of the mutually exclusive intervals D₁, D₂,...., D_m is the set of all possible values for the Y_i's, (p₁ + p₂ + .... + p_m) = 1. Define the set of discrete random variables X₁, X₂, ...., X_m, where

X₁= number of Y_i's whose valueÎD₁
X₂= number of Y_i's whose value Î D₂

:
:

X_m= number of Y_i's whose value Î D_m

and (X₁+ X₂+ .... + X_m) = n. Then the set of discrete random variables X₁, X₂, ...., X_mwill have a multinomial probability distribution with parameters n and the set of probabilities {p₁, p₂, ..., p_m}. If the intervals D₁, D₂, ...., D_m are chosen such that np_i ³ 5 for i = 1, 2, ..., m, then;

C = S (X_i - np_i) ²/ np_i.

The sum is over i = 1, 2,..., m. The results is distributed as c² _m-1.

For the goodness-of-fit sample test, we formulate the null and alternative hypothesis as

H₀ : f_Y(y) = f_o(y)
H_a : f_Y(y) ¹ f_o(y)

At the a level of significance, H₀ will be rejected in favor of H_a if

C = S (X_i - np_i) ²/ np_i

is greater than c² _m

However, it is possible that in a goodness-of-fit test, one or more of the parameters of f_o(y) are unknown. Then the probability values p₁, p₂, ..., p_m will have to be estimated by assuming that H₀ is true and calculating their estimated values from the sample data. That is, another set of probability values p'₁, p'₂, ..., p'_mwill need to be computed so that the values (np'₁, np'₂, ..., np'_m) are the estimated expected values of the multinomial random variable (X₁, X₂, ...., X_m). In this case, the random variable C will still have a Chi-square distribution, but its degrees of freedom will be reduced. In particular, if the probability function f_o(y) has r unknown parameters,

C = S (X_i - np_i) ²/ np_i

is distributed as c² _m-1-r.

For this goodness-of-fit test, we formulate the null and alternative hypothesis as

H₀: f_Y(y) = f_o(y)
H_a: f_Y(y) ¹ f_o(y)

At the a level of significance, H₀ will be rejected in favor of H_a if C is greater than c² _m-1-r.

An Application: A die is thrown 300 times and the following frequencies are observed. Test the hypothesis that the die is fair at level 0.05. Under the null hypothesis that the die is fair, the expected frequencies are all equal to 300/6 = 50. Both the Observed (O) and Expected (E) frequencies are recorded in the following table together with the random variable Y that represents the number on each sides of the die:

Goodness-of-fit Test For Discrete Variables
Y	1	2	3	4	5	6
O	57	43	59	55	63	23
E	50	50	50	50	50	50

The quantity

c ² = S [(O - E )² / E] = 22.04

is a measure of the goodness-of-fit. If there is a reasonably good fit to the hypothetical distribution, this measure will be very close to zero. Since c ² _{n-1, 0.95} = 11.07, we reject the null hypothesis that the die is a fair one.

You might like to use this JavaScript to perform this test.

For statistical equality of two random variables characterizing two populations, you might like to use the Kolmogorov-Smirnov Test if you have two independent sets of random observations, one from each population.

---

Compatibility of Multi-Counts Test

In some applications, such as quality control, it is necessary to check if the process is under control. This can be done by testing if there are significant differences between number of "counts", taken over k equal-periods of times. The counts are supposed to have been obtained under comparable conditions.

The null hypothesis is:

H₀: There is no significant difference between number of "counts" taken over k equal-periods of times.

Under the null hypothesis, the statistic:

S (N_i - N)²/N

has a Chi-square distribution with d.f. = k-1. Where i is the count's number, N_i is its counts, and N = SN_i/k.

One may extend this useful test to where the duration of obtaining the i^th count is t_i. Then the above test statistic becomes:

S [(N_i - t_iN)²/ t_iN]

and has a Chi-square distribution with d.f. = k-1, where i is the count's number, N_i is its counts, and N = SN_i/St_i.

You might like to use the Compatibility of Multi-Counts JavaScript to check your computations, and to perform some numerical experimentation for a deeper understanding of the concepts.

---

Necessary Conditions for the Above Chi-square Based Testing

Like any statistical test procedures, the Chi-square based testing must meet certain necessary conditions to apply; otherwise, any obtained conclusion might be wrong or misleading. This is true in particular for using the Chi-square-based test for cross-tabulated data.

Necessary conditions for the Chi-square based tests for crosstable data are:

1. Expected values greater than 5 in 80% or more of the cells.
2. Moreover, if number of cells is fewer than 5, then all expected values must be greater than 5.

An Example: Suppose the monthly number of accidents reported in a factory in three eight-hour shifts is 1, 7, and 7, respectively. Are the working conditions and the exposure to risk similar for all shifts? Clearly, the answer must be, No they are not. However, applying the goodness-of-fit, at 0.05, under the null hypothesis that there are no differences in the number of accidents in three shifts, one expects 5, 5, and 5 accidents in each shift. The Chi-square test statistic is:

c ² = S [(O - E )² / E] = 4.8

However, since c ² _{n-1, 0.95} = 5.99, there is no reason to reject that there is no difference, which is a very strange conclusion. What is wrong with this application?

You might like to use this JavaScript to verify your computation.

---

Testing the Variance: Is the Quality that Good?

Suppose a population has a normal distribution. The manager is to test a specific claim made about the quality of the population by way of testing its variance s². Among three possible scenarios, the interesting case is in testing the following null hypothesis based on a set of n random sample observations:

H₀: Variation is about the claimed value.
H_a: The variation is more than what is claimed, indicating the quality is much lower than expected.

Upon computing the estimated variance S² based on n observations, then the statistic:

c^½ = [(n-1). s²] / s²

has a Chi-square distribution with degree of freedom n = n - 1. This statistic is then used for testing the above null hypothesis.

You might like to use Testing the Variance JavaScript to check your computations.

---

Testing the Equality of Multi-Variances

The equality of variances across populations is called homogeneity of variances or homoscedasticity. Some statistical tests, such as testing equality of the means by the t-test and ANOVA, assume that the data come from populations that have the same variance, even if the test rejects the null hypothesis of equality of population means. If this condition of homogeneity of variance is not met, the statistical test results may not be valid. Heteroscedasticity refers to lack of homogeneity of variances.

Bartlett's Test is used to test if k samples have equal variances. It compares the Geometric Mean of the group variances to the arithmetic mean; therefore, it is a Chi-square statistic with (k-1) degrees of freedom, where k is the number of categories in the independent variable. The test is sensitive to departures from normality. The sample sizes do not have to be equal but each must be at least 6. Just like the two population t-test, ANOVA can go wrong when the equality of variances condition is not met.

The Bartlett test statistic is designed to test for equality of variances across groups against the alternative that variances are unequal for at least two groups. Formally,

H₀: All variances are almost equal.

The test statistic:

B = {S [(n_i -1)LnS²] S [(n_i -1)LnS_i²]}/ C

In the above, S_i² is the variance of the ith group, n_i is the sample size of the i^th group, k is the number of groups, and S² is the pooled variance. The pooled variance is a weighted average of the group variances and is defined as:

S² = {S [(n_i -1)S_i²]} / S [(n_i -1)], over all i = 1, 2,..,k

and

C = 1 + {S [1/(n_i -1)] - 1/ S [1/(n_i -1)] }/[3(k+1)].

You might like to use the Equality of Multi-Variances JavaScript to check your computations, and to perform some numerical experimentation for a deeper understanding of the concepts.

Rule of 2: For 3 or more populations, there is a practical rule known as the "Rule of 2". According to this rule, one divides the highest variance of a sample by the lowest variance of the other sample. Given that the sample sizes are almost the same, and the value of this division is less than 2, then, the variations of the populations are almost the same.

Example: Consider the following three random samples from three populations, P1, P2, P3:

	Sample P1	Sample P2	Sample P3
	25	17	8
	25	21	10
	20	17	14
	18	25	16
	13	19	12
	6	21	14
	5	15	6
	22	16	16
	25	24	13
	10	23	6
N	10	10	10
Mean	16.90	19.80	11.50
Std.Dev.	7.87	3.52	3.81
SE Mean	2.49	1.11	1.20

The ANOVA Table
Sources of Variation	Sum of Squares	Degrees of Freedom	Mean Squares	F-Statistic
Between Samples	79.40	2	39.70	4.38
Within Samples	244.90	27	9.07
Total	324.30	29

With an F = 4.38 and a p-value of 0.023, we reject the null at a = 0.05. This is not good news, since ANOVA, like the two-sample t-test, can go wrong when the equality of variances condition is not met.

Further Readings:
Hand D., and C. Taylor, Multivariate Analysis of Variance and Repeated Measures, Chapman and Hall, 1987.
Miller R. Jr, Beyond ANOVA: Basics of Applied Statistics, Wiley, 1986.

---

Correlation Coefficients Testing

The Fisher's Z-transformation is a useful tool in the circumstances in which two or more independent correlation coefficients are to be compared simultaneously. To perform such a test one may evaluate the Chi-square statistic:

c² = S[(n_i - 3).Z_i²] - [S(n_i - 3).Z_i]² / [S(n_i - 3)],    the sums are over all i = 1, 2, .., k.

Where the Fisher Z-transformation is

Z_i = 0.5[Ln(1+r_i) - Ln(1-r_i)],   provided | r_i | ¹ 1.

Under the null hypothesis:

H₀: All correlation coefficients are almost equal.

The test statistic c² has (k-1) degrees of freedom, where k is the number of populations.

An Application: Consider the following correlation coefficients obtained by random sampling form ten independent populations.

Population Pi Correlation ri Sample Size ni 1 0.72 67 2 0.41 93 3 0.57 73 4 0.53 98 5 0.62 82 6 0.21 39 7 0.68 91 8 0.53 27 9 0.49 75 10 0.50 49

Using the above formula c²-statistic = 19.916, that has a p-value of 0.02. Therefore, there is moderate evidence against the null hypothesis.

In such a case, one may omit a few outliers from the group, then use the Test for Equality of Several Correlation Coefficients JavaScript. Repeat this process until a possible homogeneous sub-group may emerge.

You might need to use Sample Size Determination JavaScript at the design stage of your statistical investigation in decision making with specific subjective requirements.

---

Simple Linear Regression: Computational Aspects

The regression analysis has three goals: predicting, modeling, and characterization. What would be the logical order in which to tackle these three goals such that one task leads to and /or and justifies the other tasks? Clearly, it depends on what the prime objective is. Sometimes you wish to model in order to get better prediction. Then the order is obvious. Sometimes, you just want to understand and explain what is going on. Then modeling is again the key, though out-of-sample predicting may be used to test any model. Often modeling and predicting proceed in an iterative way and there is no 'logical order' in the broadest sense. You may model to get predictions, which enable better control, but iteration is again likely to be present and there are sometimes special approaches to control problems.

The following contains the main essential steps during modeling and analysis of regression model building, presented in the context of an applied numerical example.

Formulas and Notations:

• = Sx /n
  This is just the mean of the x values.
• = Sy /n
  This is just the mean of the y values.
• S_xx = SS_xx = S(x(i) - )² = Sx² - ( Sx)² / n
• S_yy = SS_yy = S(y(i) - )² = Sy² - ( Sy) ² / n
• S_xy = SS_xy = S(x(i) - )(y(i) - ) = Sx ×y – (Sx) × (Sy) / n
• Slope m = SS_xy / SS_xx
• Intercept, b = - m .
• y-predicted = yhat(i) = m×x(i) + b.
• Residual(i) = Error(i) = y – yhat(i).
• SSE = S_res = SS_res = SS_errors = S[y(i) – yhat(i)]².
• Standard deviation of residuals = s = S_res = S_errors = [SS_res / (n-2)]^1/2.
• Standard error of the slope (m) = S_res / SS_xx^1/2.
• Standard error of the intercept (b) = S_res[(SS_xx + n. ²) /(n × SS_xx] ^1/2.

A computational Example: A taxicab company manager believes that the monthly repair costs (Y) of cabs are related to age (X) of the cabs. Five cabs are selected randomly and from their records we obtained the following data: (x, y) = {(2, 2), (3, 5), (4, 7), (5, 10), (6, 11)}. Based on our practical knowledge and the scattered diagram of the data, we hypothesize a linear relationship between predictor X, and the cost Y.

Now the question is how we can best (i.e., least square) use the sample information to estimate the unknown slope (m) and the intercept (b)? The first step in finding the least square line is to construct a sum of squares table to find the sums of x values (Sx), y values (Sy), the squares of the x values (Sx²), the squares of the x values (Sy²), and the cross-product of the corresponding x and y values (Sxy), as shown in the following table:

	x	y	x2	xy	y2
	2	2	4	4	4
	3	5	9	15	25
	4	7	16	28	49
	5	10	25	50	100
	6	11	36	66	121
SUM	20	35	90	163	299

The second step is to substitute the values of Sx, Sy, Sx², Sxy, and Sy² into the following formulas:

SS_xy = Sxy – (x)(Sy)/n = 163 - (20)(35)/5 = 163 - 140 = 23

SS_xx = Sx² – (Sx)²/n = 90 - (20)²/5 = 90- 80 = 10

SS_yy = Sy² – (Sy)²/n = 299 - 245 = 54

Use the first two values to compute the estimated slope:

Slope = m = SS_xy / SS_xx = 23 / 10 = 2.3

To estimate the intercept of the least square line, use the fact that the graph of the least square line always pass through (, ) point, therefore,

The intercept = b = – (m)() = (Sy)/ 5 – (2.3) (Sx/5) = 35/5 – (2.3)(20/5) = -2.2

Therefore the least square line is:

y-predicted = yhat = mx + b = -2.2 + 2.3x.

After estimating the slope and the intercept the question is how we determine statistically if the model is good enough, say for prediction. The standard error of slope is:

Standard error of the slope (m)= S_m = S_res / S_xx^1/2,

and its relative precision is measured by statistic

t_slope = m / S_m.

For our numerical example, it is:

t_slope = 2.3 / [(0.6055)/ (10^1/2)] = 12.01

which is large enough, indication that the fitted model is a "good" one.

You may ask, in what sense is the least squares line the "best-fitting" straight line to 5 data points. The least squares criterion chooses the line that minimizes the sum of square vertical deviations, i.e., residual = error = y - yhat:

SSE = S (y – yhat)² = S(error)² = 1.1

The numerical value of SSE is obtained from the following computational table for our numerical example.

	x Predictor	-2.2+2.3x y-predicted	y observed	error y	squared errors
	2	2.4	2	-0.4	0.16
	3	4.7	5	0.3	0.09
	4	7	7	0	0
	5	9.3	10	0.7	0.49
	6	11.6	11	-0.6	0.36
				Sum=0	Sum=1.1

Alternately, one may compute SSE by:

SSE = SS_yy – m SS_xy = 54 – (2.3)(23) = 54 - 52.9 = 1.1,

as expected

Notice that this value of SSE agrees with the value directly computed from the above table. The numerical value of SSE gives the estimate of variation of the errors s²:

s² = SSE / (n -2) = 1.1 / (5 - 2) = 0.36667

The estimate the value of the error variance is a measure of variability of the y values about the estimated line. Clearly, we could also compute the estimated standard deviation s of the residuals by taking the square roots of the variance s².

As the last step in the model building, the following Analysis of Variance (ANOVA) table is then constructed to assess the overall goodness-of-fit using the F-statistics:

Analysis of Variance Components
Source	DF	Sum of Squares	Mean Square	F Value	Prob > F
Model	1	52.90000	52.90000	144.273	0.0012
Error	3	SSE = 1.1	0.36667
Total	4	SSyy = 54

For practical proposes, the fit is considered acceptable if the F-statistic is more than five-times the F-value from the F distribution tables at the back of your textbook. Note that, the criterion that the F-statistic must be more than five-times the F-value from the F distribution tables is independent of the sample size.

Notice also that there is a relationship between the two statistics that assess the quality of the fitted line, namely the T-statistics of the slope and the F-statistics in the ANOVA table. The relationship is:

t²_slope = F

This relationship can be verified for our computational example.

Predictions by Regression: After we have statistically checked the goodness of-fit of the model and the residuals conditions are satisfied, we are ready to use the model for prediction with confidence. Confidence interval provides a useful way of assessing the quality of prediction. In prediction by regression often one or more of the following constructions are of interest:

1. A confidence interval for a single future value of Y corresponding to a chosen value of X.
2. A confidence interval for a single pint on the line.
3. A confidence region for the line as a whole.

Confidence Interval Estimate for a Future Value: A confidence interval of interest can be used to evaluate the accuracy of a single (future) value of y corresponding to a chosen value of X (say, X₀). This JavaScript provides confidence interval for an estimated value Y corresponding to X₀ with a desirable confidence level 1 - a.

Y_p ± S_e . t_{n-2, a/2} {1/n + (X₀ – )²/ S_x}^1/2

Confidence Interval Estimate for a Single Point on the Line: If a particular value of the predictor variable (say, X₀) is of special importance, a confidence interval on the value of the criterion variable (i.e. average Y at X₀) corresponding to X₀ may be of interest. This JavaScript provides confidence interval on the estimated value of Y corresponding to X₀ with a desirable confidence level 1 - a.

Y_p ± S_e . t_{n-2, a/2} { 1 + 1/n + (X₀ – )²/ S_x}^1/2

It is of interest to compare the above two different kinds of confidence interval. The first kind has larger confidence interval that reflects the less accuracy resulting from the estimation of a single future value of y rather than the mean value computed for the second kind confidence interval. The second kind of confidence interval can also be used to identify any outliers in the data.

Confidence Region the Regression Line as the Whole: When the entire line is of interest, a confidence region permits one to simultaneously make confidence statements about estimates of Y for a number of values of the predictor variable X. In order that region adequately covers the range of interest of the predictor variable X; usually, data size must be more than 10 pairs of observations.

Y_p ± S_e { (2 F_{2, n-2, a}) . [1/n + (X₀ – )²/ S_x]}^1/2

In all cases the JavaScript provides the results for the nominal (x) values. For other values of X one may use computational methods directly, graphical method, or using linear interpolations to obtain approximated results. These approximation are in the safe directions i.e., they are slightly wider that the exact values.

---

Regression Modeling and Analysis

Many problems in analyzing data involve describing how variables are related. The simplest of all models describing the relationship between two variables is a linear, or straight-line, model. Linear regression is always linear in the coefficients being estimated, not necessarily linear in the variables.

The simplest method of drawing a linear model is to "eye-ball" a line through the data on a plot, but a more elegant, and conventional method is that of least squares, which finds the line minimizing the sum of the vertical distances between observed points and the fitted line. Realize that fitting the "best" line by eye is difficult, especially when there is much residual variability in the data.

Know that there is a simple connection between the numerical coefficients in the regression equation and the slope and intercept of the regression line.

Know that a single summary statistic, like a correlation coefficient, does not tell the whole story. A scatterplot is an essential complement to examining the relationship between the two variables.

Again, the regression line is a group of estimates for the variable plotted on the Y-axis. It has a form of y = b + mx, where m is the slope of the line. The slope is the rise over run. If a line goes up 2 for each 1 it goes over, then its slope is 2.

The regression line goes through a point with coordinates of (mean of x values, mean of y values), known as the mean-mean point.

If you plug each x in the regression equation, then you obtain a predicted value for y. The difference between the predicted y and the observed y is called a residual, or an error term. Some errors are positive and some are negative. The sum of squares of the errors plus the sum of squares of the estimates add up to the sum of squares of Y. The regression line is the line that minimizes the variance of the errors. The mean error is zero; so, this means that it minimizes the sum of the squares errors.

The reason for finding the best fitting line is so that you can make a reasonable prediction of what y will be if x is known (not vise-versa).

r² is the variance of the estimates divided by the variance of Y. r is the size of the slope of the regression line, in terms of standard deviations. In other words, it is the slope of the regression line if we use the standardized X and Y. It is how many standard deviations of Y you would go up, when you go one standard deviation of X to the right.

Coefficient of Determination: Another measure of the closeness of the points to the regression line is the Coefficient of Determination:

r² = SS_{^{yhat yhat}} / SS_yy

which is the amount of the squared deviation in Y, that is explained by the points on the least squares regression line.

Homoscedasticity and Heteroscedasticity: Homoscedasticity (homo = same, skedasis = scattering) is a word used to describe the distribution of data points around the line of best fit. The opposite term is heteroscedasticity. Briefly, homoscedasticity means that data points are distributed equally about the line of best fit. Therefore, homoscedasticity means constancy of variance over all the levels of factors. Heteroscedasticity means that the data points cluster or clump above and below the line in a non-equal pattern.

Standardized Regression Analysis: The scale of measurements used to measure X and Y has major impact on the regression equation and correlation coefficient. This impact is more drastic comparing two regression equations having different scales of measurement. To overcome these drawbacks, one must standardize both X and Y prior to constructing the regression and interpreting the results. In such a model, the slope is equal to the correlation coefficient r. Notice that the derivative of function Y with respect to dependent variable X is the correlation coefficient. Therefore, there is a nice similarity in the meaning of r in statistics and the derivative from calculus, in that its sign and its magnitude reveal the increasing/decreasing and the rate of change, as the derivative of a function do.

In the usual regression modeling the estimated slope and intercept are correlated; therefore, any error in estimating the slope influences the estimate of the intercept. One of the main advantages of using the standardized data is that the intercept is always equal to zero.

Regression when both X and Y are random: Simple linear least-squares regression has among its conditions that the data for the independent (X) variables are known without error. In fact, the estimated results are conditioned on whatever errors happened to be present in the independent data sets. When the X-data have an error associated with them the result is to bias the slope downwards. A procedure known as Deming regression can handle this problem quite well. Biased slope estimates (due to error in X) can be avoided using Deming regression.

If X and Y are random variables, then the correlation coefficient R is often referred to as the Coefficient of Reliability.

The Relationship Between Slope and Correlation Coefficient: By a little bit of algebraic manipulation, one can show that the coefficient of correlation is related to the slope of the two regression lines: Y on X, and X on Y, denoted by m _yx and m_xy, respectively:

R² = m _yx . m_xy

Lines of regression through the origin: Often the conditions of a practical problem require that the regression line go through the origin (x = 0, y = 0). In such a case, the regression line has one parameter only, which is its slope:

m = S (x_i ´ y_i)/ Sx_i²

Notice that, for the models with the omission of the intercept, it is generally agreed that R² should not be defined or even considered.

Parabola models: Parabola regressions have three coefficients with a general form:

Y = a + bX + cX²,

where

c = { S (x_i - xbar)²×y_i - n[S(x_i - xbar) ²× Sy_i]} / {n S(x_i - xbar) ⁴ - [S(x_i - xbar)²] ²}

b = [S(x_i- xbar) y_i]/[ S(x_i - xbar)²] - 2×c×xbar

a = {Sy_i - [c× S(x _i - xbar) ²)}/n - (c×xbar×xbar + b×xbar),

where xbar is the mean of x_i's.

Applications of quadratic regression include fitting the supply and demand curves in econometrics and fitting the ordering cost and holding cost functions in inventory control for finding the optimal ordering quantity.

You might like to use Quadratic Regression JavaScript to check your hand computation. For higher degrees than quadratic, you may like to use the Polynomial Regressions JavaScript.

Multiple Linear Regression: The objectives in a multiple regression problem are essentially the same as for a simple regression. While the objectives remain the same, the more predictors we have the calculations and interpretations become more complicated. With multiple regression, we can use more than one predictor. It is always best, however, to be parsimonious, that is to use as few variables as predictors as necessary to get a reasonably accurate forecast. Multiple regression is best modeled with commercial package such as SAS and SPSS. The forecast takes the form:

Y = b₀ + b₁X₁ + b₂X₂ + . . .+ b_nX_n,

where b₀ is the intercept, b₁, b₂, . . . b_n are coefficients representing the contribution of the independent variables X₁, X₂,..., X_n.

For small sample size, you may like to use the Multiple Linear Regression JavaScript.

What Is Auto-Regression: In time series analysis and forecasting techniques, often linear regression is use to combine present and past values of an observation in order to forecast its future value. The model is called an autoregressive model. For details and implementation process visit Autoregressive Modeling JavaScript.

What Is Logistic Regression: Standard logistic regression is a method for modeling binary data (e.g., does a person smoke or not?, does a person survive a disease, or not?). Polygamous logistic regression is a method for modeling more than two options (e.g., does a person take the bus, drive a car or take the subway? does an office use WordPerfect, Word, or other office-ware?).

Why Linear Regression? The study of corn shell (i.e., ear of corn) height versus rainfall has shown to have the following regression curve:

Clearly, the relationship is highly nonlinear; however, if we are interested in a "small" range (say, for a specific geographical area, like southern region of the state of Maryland) then the condition of linearity might be satisfactory. A typical application is depicted in the above figure where we are interested in predicting the height of corn in an area with rainfall in the range of [a, b]. Magnifying process of scale for this range allows us to fit a useful linear regression. If the range is not short enough, then one may sub-divide the range accordingly by applying the same process of fitting a few lines, one for each sub-interval.

Structural Changes: When a regression model has been estimated using the available data set, an additional data set may sometimes become available. To test if previous model is still valid or the two separate models are equivalent or not, one may use the analysis of covariance testing described on this site.

You might like to use the Regression Analysis JavaScript to check your computations and to perform some numerical experimentation for a deeper understanding of the concepts.

Further Reading:
Chatterjee S., B. Price, and A. Hadi, Regression Analysis by Example, Wiley, 1999.

---

Regression Modeling Selection Process

When you have more than one regression equation based on data, to select the "best model", you should compare: