You might like to use the Descriptive Sampling Statistics JavaScript in performing some numerical experimentation for validating the above assertions for a deeper understanding.

---

Shape of a Distribution Function:
The Skewness-Kurtosis Chart

The pair of statistical measures, skewness and kurtosis, are measuring tools, which is used in selecting a distribution(s) to fit your data. To make an inference with respect to the population distribution, you may first compute skewness and kurtosis from your random sample from the entire population. Then, locating a point with these coordinates on the widely used skewness-kurtosis chart , guess a couple of possible distributions to fit your data. Finally, you might use the goodness-of-fit test to rigorously come up with the best candidate fitting your data. Removing outliers improves the accuracy of both skewness and kurtosis.

Skewness: Skewness is a measure of the degree to which the sample population deviates from symmetry with the mean at the center.

Skewness = S (x_i - ) ³ / [ (n - 1) S ³ ],    n is at least 2.

Skewness will take on a value of zero when the distribution is a symmetrical curve. A positive value indicates the observations are clustered more to the left of the mean with most of the extreme values to the right of the mean. A negative skewness indicates clustering to the right. In this case we have: Mean £ Median £ Mode. The reverse order holds for the observations with positive skewness.

Kurtosis: Kurtosis is a measure of the relative peakedness of the curve defined by the distribution of the observations.

Kurtosis = S (x_i - ) ⁴ / [ (n - 1) S ⁴ ],     n is at least 2.

Standard normal distribution has kurtosis of +3. A kurtosis larger than 3 indicates the distribution is more peaked than the standard normal distribution.

Coefficient of Excess Kurtosis = Kurtosis - 3.

A value of less than 3 for kurtosis indicates that the distribution is flatter than the standard normal distribution.

It can be shown that,

Kurtosis - Skewness ² is greater than or equal to 1, and
    Kurtosis is less than or equal to the sample size n.

These inequalities hold for any probability distribution having finite skewness and kurtosis.

In the Skewness-Kurtosis Chart, you notice two useful families of distributions, namely the beta and gamma families.

The Beta-Type Density Function: Since the beta density has both a shape and a scale parameter, it describes many random phenomena provided the random variable is between [0, 1]. For example, when both parameters are integer with random variables the result is the binomial Probability function.

Applications: A basic distribution of statistics for variables bounded at both sides; for example x between [0, 1]. The beta density is useful for both theoretical and applied problems in many areas. Examples include distribution of proportion of population located between lowest and highest value in sample; distribution of daily per cent yield in a manufacturing process; description of elapsed times to task completion (PERT). There is also a relationship between the Beta and Normal distributions. The conventional calculation is that given a PERT Beta with highest value as b, lowest as a, and most likely as m, the equivalent normal distribution has a mean and mode of (a + 4m + b)/6 and a standard deviation of (b - a)/6.

Comments: Uniform, right triangular, and parabolic distributions are special cases. To generate beta, generate two random values from a gamma, g₁, g₂. The ratio g₁/(g₁ +g₂) is distributed like a beta distribution. The beta distribution can also be thought of as the distribution of X1 given (X1+X2), when X1 and X2 are independent gamma random variables.

Gamma-Type Density Function: Some random variables are always non-negative. The density function associated with these random variables often is adequately modeled as the gamma density function. The Gamma-Type Density Function has both a shape and a scale parameter. With both the shape and scale parameters equal to 1, the result is the exponential density function. Chi-square is also a special case of gamma density function with shape parameter equal to 2.

Applications: A basic distribution of statistics for variables bounded at one side ; for example x greater than or equal to zero. The gamma density gives distribution of time required for exactly k independent events to occur, assuming events take place at a constant rate. Used frequently in queuing theory, reliability, and other industrial applications. Examples include distribution of time between re-calibrations of instrument that needs re-calibration after k uses; time between inventory restocking, time to failure for a system with standby components.

Comments: Erlangian, Exponential, and Chi-square distributions are special cases. The negative binomial is an analog to gamma distribution with discrete random variable.

What is the distribution of the product of sample observations from the uniform (0, 1) random? Like many problems with products, this becomes a familiar problem when turned into a problem about sums. If X is uniform (for simplicity of notation make it U(0,1)), Y=-log(X) is exponentially distributed, so the log of the product of X1, X2, ... Xn is the sum of Y1, Y2, ... Yn which has a gamma (scaled Chi-square) distribution. Thus, it is a gamma density with shape parameter n and scale 1.

The Log-normal Density Function: Permits representation of a random variable whose logarithm follows a normal distribution. The ratio of two log-normally random variables is also log-normal.

Applications: Model for a process arising from many small multiplicative errors. Appropriate when the value of an observed variable is a random proportion of the previously observed value.

Applications: Examples include distribution of sizes from a breakage process; distribution of income size, inheritances and bank deposits; distribution of various biological phenomena; life distribution of some transistor types.

The lognormal distribution is widely used in situations where values are positively skewed (where the distribution has a long right tail; negatively skewed distributions have a long left tail; a normal distribution has no skewness). Examples of data that"fit" a lognormal distribution include financial security valuations or real estate property valuations. Financial analysts have observed that the stock prices are usually positively skewed, rather than normally (symmetrically) distributed. Stock prices exhibit this trend because the stock price cannot fall below the lower limit of zero but may increase to any price without limit. Similarly, healthcare costs illustrate positive skewness since unit costs cannot be negative. For example, there can't be negative cost for services in a capitation contract. This distribution accurately describes most healthcare data.

In the case where the data are log-normally distributed, the Geometric Mean acts as a better data descriptor than the mean. The more closely the data follow a log-normal distribution, the closer the geometric mean is to the median, since the log re-expression produces a symmetrical distribution.

Further Reading:
Snell J., Introduction to Probability, Random House, 1987. Read section 4.2 for a link between beta and F distributions (with the advantage that tables are easy to find).
Tabachnick B., and L. Fidell, Using Multivariate Statistics, HarperCollins, 1996. Has a good discussion on applications and significance tests for skewness and kurtosis.

---

Numerical Example and Discussions

A Numerical Example: Given the following, small (n = 4) data set, compute the descriptive statistics: x₁ = 1, x₂ = 2, x₃ = 3, and x₄ = 6.

i	xi	( xi- )	( xi - ) 2	( xi - ) 3	( xi - )4
1	1	-2	4	-8	16
2	2	-1	1	-1	1
3	3	0	0	0	0
4	6	3	9	27	81
Sum	12	0	14	18	98

The mean is 12 / 4 = 3; the variance is s² = 14 / 3 = 4.67; the standard deviation is s = (14/3) ^0.5 = 2.16; the skewness is 18 / [3 (2.16) ³ ] = 0.5952, and finally, the kurtosis is 98 / [3 (2.16) ⁴] = 1.5.

You might like to use Descriptive Statistics to check your hand computation.

A Short Discussion on the Descriptive Statistic:

Deviations about the mean m of a distribution is the basis for most of the statistical tests we will learn. Since we are measuring how much a set of scores is dispersed about the mean m , we are measuring variability. We can calculate the deviations about the mean m and express it as variance s² or standard deviation s. It is very important to have a firm grasp of this concept because it will be a central concept throughout your statistics course.

Both variance s² and standard deviation s measure variability within a distribution. Standard deviation s is a number that indicates how much on average each of the values in the distribution deviates from the mean m (or center) of the distribution. Keep in mind that variance s² measures the same thing as standard deviation s (dispersion of scores in a distribution). Variance s², however, is the average squared deviations about the mean m . Thus, variance s² is the square of the standard deviation s.

The expected value and the variance of the statistic are m and s²/n, respectively.

The expected value and variance of statistic S² are s² and 2s⁴ / (n-1), respectively.

and S² are the best estimators for m and s². They are Unbiased (you may update your estimate); Efficient (they have the smallest variation among other estimators); Consistent (increasing sample size provides a better estimate); and Sufficient (you do not need to have the whole data set; what you need are Sx_i and Sx_i² for estimations). Note also that the above variance S² is justified only in the case where the population distribution tends to be normal, otherwise one may use bootstrapping techniques.

In general, it is believed that the pattern of mode, median, and mean go from lower to higher in positive skewed data sets, and just the opposite pattern in negative skewed data sets. However; for example, in the following 23 numbers, mean = 2.87, median = 3, but the data is positively skewed:

4, 2, 7, 6, 4, 3, 5, 3, 1, 3, 1, 2, 4, 3, 1, 2, 1, 1, 5, 2, 2, 3, 1

and, the following 10 numbers have mean = median = mode = 4, but the data set is left skewed:

1, 2, 3, 4, 4, 4, 5, 5, 6, 6.

Note also, that most commercial software do not correctly compute skewness and kurtosis. There is no easy way to determine confidence intervals about a computed skewness or kurtosis value from a small to medium sample. The literature gives tables based on asymptotic methods for sample sets larger than 100 for normal distributions only.

You may have noticed that using the above numerical example on some computer packages such as SPSS, the skewness and the kurtosis are different from what we have computed. For example, the SPSS output for the skewness is 1.190. However, for large a sample size n, the results are identical.

Reference and Further Readings:
David H., Early sample measures of variability, Statistical Science, Vol. 13, 1998, 368-377. This article provides a good historical account of statistical measures.
Groeneveld R., A class of quantile measures for kurtosis, The American Statistician, 325, Nov. 1998.
Lehmann E., Testing Statistical Hypotheses, 1996, Wiley. Exact confidence interval for the coefficient of variation is computationally tedious as shown in this book.

---

The Two Statistical Representations of a Population

The following figure depicts a typical relationship between the cumulative distribution function (cdf) and the density (for continuous random variables),

All characteristics of the population are well described by either of these two functions. The figure also illustrates their applications in determining the (lower) percentile measures denoted by P:

P = P[ X £ x] = Probability that the random variable
X is less than or equal to a given number x,

among other useful information. Notice that the probability P is the area under the density function curve, while numerically equal to the height of cdf curve at point x.

Both functions can be estimated by smoothing the empirical (i.e., observed) cumulative step-function, and smoothing the histogram constructed from a random sample.

---

Empirical (i.e., observed) Cumulative Distribution Function

The empirical cumulative distribution function (ECDF), also known as Ogive (pronounced o-jive), is used to graph cumulative frequency.

The ogive is the estimator for the population's cumulative distribution function, which contains all the characteristic of the population. The empirical distribution is a staircase function with the location of the drops randomly placed. The size of the each stair at each point depends on the frequency of that point value, and it is equal to the frequency/n where n is the sample size. The sample size is the sum of all frequencies.

Note that almost all statistics we have covered up to now can be obtained and understood more deeply by graph paper using Empirical Distribution Function JavaScript. You may like using this JavaScript in performing some numerical experimentation for a deeper understanding.

Other widely used decision model based upon empirical cumulative distribution function (ECDF) as a measuring tool and decision procedure are the ABC Inventory Classification, Single-period Inventory Analysis (The Newsboy Model), and determination of the Best Time to Replace Equipment. For other inventory decisions, visit the Inventory Control Models site.

---

Introduction

Modeling of a Data Set: Families of parametric distribution models are widely used to summarize a huge data set, to obtain predictions, assess goodness of fit, to estimate functions of the data not easily derived directly, or to render manageable random effects. The trustworthiness of the results obtained depends on the generality of the distribution family employed.

Inductive Inference: This extension of our knowledge from a particular random sample to the population is called inductive inference. The main function of business statistics is the provision of techniques for making inductive inference and for measuring the degree of uncertainty of such inference. Uncertainty is measured in terms of probability statements, and that is the reason we need to learn the language of uncertainty and its measuring tool called probability.

In contrast to the inductive inference, mathematics often uses deductive inference to prove theorems, while in empirical science, such as statistics, inductive inference is used to find new knowledge or to extend our knowledge.

Further Readings:
Brown B., F. Spears, and L. Levy, The log F: A distribution for all seasons, Computational Statistics, 17(1), 47-58, 2002.

---

Probability, Chance, Likelihood, and Odds

The concept of probability occupies an important place in the decision-making process under uncertainty, whether the problem is one faced in business, in government, in the social sciences, or just in one's own everyday personal life. In very few decision-making situations is perfect information -- all the needed facts -- available. Most decisions are made in the face of uncertainty. Probability enters into the process by playing the role of a substitute for certainty - a substitute for complete knowledge.

Probability is especially significant in the area of statistical inference. Here the statistician's prime concern lies in drawing conclusions or making inferences from experiments which involve uncertainties. The concepts of probability make it possible for the statistician to generalize from the known (sample) to the unknown (population) and to place a high degree of confidence in these generalizations. Therefore, Probability is one of the most important tools of statistical inference.

Probability has an exact technical meaning -- well, in fact it has several, and there is still debate as to which term ought to be used. However, for most events for which probability is easily computed; e.g., rolling of a die, the probability of getting a four [::], almost all agree on the actual value (1/6), if not the philosophical interpretation. A probability is always a number between 0 and 1. Zero is not"quite" the same thing as impossibility. It is possible that"if" a coin were flipped infinitely many times, it would never show"tails", but the probability of an infinite run of heads is 0. One is not"quite" the same thing as certainty but close enough.

The word"chance" or"chances" is often used as an approximate synonym of "probability", either for variety or to save syllables. It would be better practice to leave"chance" for informal use, and say"probability" if that is what is meant. One occasionally sees"likely" and"likelihood"; however, these terms are used casually as synonyms for"probable" and"probability".

Odds is a probabilistic concept related to probability. It is the ratio of the probability (p) of an event to the probability (1-p) that it does not happen: p/(1-p). It is often expressed as a ratio, often of whole numbers; e.g.,"odds" of 1 to 5 in the die example above, but for technical purposes the division may be carried out to yield a positive real number (here 0.2). Odds are a ratio of nonevents to events. If the event rate for a disease is 0.1 (10 per cent), its nonevent rate is 0.9 and therefore its odds are 9:1.

Another way to compare probabilities and odds is using"part-whole thinking" with a binary (dichotomous) split in a group. A probability is often a ratio of a part to a whole; e.g., the ratio of the part [those who survived 5 years after being diagnosed with a disease] to the whole [those who were diagnosed with the disease]. Odds are often a ratio of a part to a part; e.g., the odds against dying are the ratio of the part that succeeded [those who survived 5 years after being diagnosed with a disease] to the part that 'failed' [those who did not survive 5 years after being diagnosed with a disease].

Aside from their value in betting, odds allow one to specify a small probability (near zero) or a large probability (near one) using large whole numbers (1,000 to 1 or a million to one). Odds magnify small probabilities (or large probabilities) so as to make the relative differences visible. Consider two probabilities: 0.01 and 0.005. They are both small. An untrained observer might not realize that one is twice as much as the other. But if expressed as odds (99 to 1 versus 199 to 1) it may be easier to compare the two situations by focusing on large whole numbers (199 versus 99) rather than on small ratios or fractions.

---

How to Assign Probabilities?

Probability is an instrument to measure the likelihood of the occurrence of an event. There are five major approaches of assigning probability: Classical Approach, Relative Frequency Approach, Subjective Approach, Anchoring, and the Delphi Technique:

1. Classical Approach: Classical probability is predicated on the condition that the outcomes of an experiment are equally likely to happen. The classical probability utilizes the idea that the lack of knowledge implies that all possibilities are equally likely. The classical probability is applied when the events have the same chance of occurring (called equally likely events), and the sets of events are mutually exclusive and collectively exhaustive. The classical probability is defined as:
   P(X) = Number of favorable outcomes / Total number of possible outcomes

2. Relative Frequency Approach: Relative probability is based on accumulated historical or experimental data. Frequency-based probability is defined as:
   P(X) = Number of times an event occurred / Total number of opportunities for the event to occur.

   Note that relative probability is based on the ideas that what has happened in the past will hold.

3. Subjective Approach: The subjective probability is based on personal judgment and experience. For example, medical doctors sometimes assign subjective probability to the length of life expectancy for a person who has cancer.

4. Anchoring: is the practice of assigning a value obtained from a prior experience and adjusting the value in consideration of current expectations or circumstances
5. The Delphi Technique: It consists of a series of questionnaires. Each series is one"round". The responses from the first"round" are gathered and become the basis for the questions and feedback of the second"round". The process is usually repeated for a predetermined number of"rounds" or until the responses are such that a pattern is observed. This process allows expert opinion to be circulated to all members of the group and eliminates the bandwagon effect of majority opinion.

   Delphi Analysis is used in decision making processes, in particular in forecasting. Several"experts" sit together and try to compromise on something upon which they cannot agree.

Further Reading:
Delbecq, A., Group Techniques for Program Planning, Scott Foresman, 1975.

---

General Computational Probability Rules

1. Addition: When two or more events will happen at the same time, and the events are not mutually exclusive, then:
   P (X or Y) = P (X) + P (Y) - P (X and Y)

   Notice that, the equation P (X or Y) = P (X) + P (Y) - P (X and Y), contains especial events: An event (X and Y) which is the intersection of set/events X and Y, and another event (X or Y) which is the union (i.e., either/or) of sets X and Y. Although this is very simple, it says relatively little about how event X influences event Y and vice versa. If P (X and Y) is 0, indicating that events X and Y do not intersect (i.e., they are mutually exclusive), then we have P (X or Y) = P (X) + P (Y). On the other hand if P (X and Y) is not 0, then there are interactions between the two events X and Y. Usually it could be a physical interaction between them. This makes the relationship P (X or Y) = P (X) + P (Y) - P (X and Y) nonlinear because the P(X and Y) term is subtracted from which influences the result.

   The above rule is known also as the Inclusion-Exclusion Formula. It can be extended to more than two events. For example, for three events A, B, and C, it becomes:

   P(A or B or C) =
   P(A) + P(B) + P(C) - P(A and B) - P(A and C) - P(B and C) + P(A and B and C)

2. Special Case of Addition: When two or more events will happen at the same time, and the events are mutually exclusive, then:
   P(X or Y) = P(X) + P(Y)

3. General Multiplication Rule: When two or more events will happen at the same time, and the events are dependent, then the general rule of multiplicative rule is used to find the joint probability:
   P(X and Y) = P(Y) ´ P(X|Y),

   where P(X|Y) is a conditional probability.

4. Special Case of Multiplicative Rule: When two or more events will happen at the same time, and the events are independent, then the special rule of multiplication rule is used to find the joint probability:
   P(X and Y) = P(X) ´ P(Y)

5. Conditional Probability: A conditional probability is denoted by P(X|Y). This phrase is read: the probability that X will occur given that Y is known to have occurred.

   Conditional probabilities are based on knowledge of one of the variables. The conditional probability of an event, such as X, occurring given that another event, such as Y, has occurred is expressed as:

   P(X|Y) = P(X and Y) ¸ P(Y),

   provided P(Y) is not zero. Note that when using the conditional rule of probability, you always divide the joint probability by the probability of the event after the word given. Thus, to get P(X given Y), you divide the joint probability of X and Y by the unconditional probability of Y. In other words, the above equation is used to find the conditional probability for any two dependent events.

   The simplest version of the Bayes' Theorem is:

   P(X|Y) = P(Y|X) ´ P(X) ¸ P(Y)

   If two events, such as X and Y, are independent then:

   P(X|Y) = P(X),
   and P(Y|X) = P(Y)

6. The Bayes' Rule:
   P(X|Y) = [ P(X) ´ P(Y|X) ] ¸ [P(X) ´P(Y|X) + P(not X) ´ P(Y| not X)]

   Bayes' rule provides posterior probability [i.e, P(X|Y)] sharpening the prior probability [i.e., P(X)] by the availability of accurate and relevant information in probabilistic terms.

An Application: Suppose two machines, A and B, produce identical parts. Machine A has probability 0.1 of producing a defective each time, whereas Machine B has probability 0.4 of producing a defective. Each machine produces one part. One of these parts is selected at random, tested, and found to be defective. What is the probability that it was produced by Machine B?

Probability tree diagrams depict events or sequences of events as branches of a tree. Tree diagrams are useful for visualizing the conditional probabilities:

The probabilities at the end of each branch are the probability that events leading to that end will happen simultaneously. The above tree diagram indicates that the probability of a part testing Good is 9/20 + 6/20 = 3/4, therefore the probability of Bad is 1/4. Thus, P(made by B | it is bad) = (4/20) / (1/4) = 4/5.

Now using the Bayes' Rule we are able to obtain useful information such as:

P(it is bad | made by B) = 1/4(4/5) / [1/4(4/5) + 3/4(2/5)] = 2/5.

Equivalently, using the above conditional probability, results in:

P(it is bad | made by B) = P(it is bad & made by B)/P(made by B) = (4/20)/(1/2) = 2/5.

Venn Diagram: A diagram used, in general to represent sets and subsets. It is a way of displaying how different sets of objects overlap. John Venn an English mathematician devised them. Venn diagram could be used as a computational probability tool similar to the probability tree diagram. The following are Venn diagrams representation for two of the above Probability Rules:

   

An Application: A surveys show that 70% of all convenience store shoppers buy milk and 55% buy bread. If 45% buy both bread and milk, what percentage buy neither?

Solution: The Venn diagram model for this problem is depicted below:

The solution is readily available from the above Venn diagram model, i.e.

P [buy neither] = 1 - [0.25 + 0.45 + 0.1] = 20%

Another approach is to use both, first the Complement Probability Rule and then the Addition Probability Rule, i.e.

P [buy neither] = 1 - P[bread OR milk] = 1 - [0.70 + 0.55 – 0.45] = 20%

It is up to you to decide which approach is “nicer” and more transparent.

Exercise Your Knowledge on the following probabilistic problem: An urn contains 4 red-balls (representing, say defective items) and 8 white-balls (Representing, say non-defective items), as depicted below:

An Urn Model

Suppose 2 balls are drawn at random. Use the following tree diagram, which is a probabilistic model for this experiment, and verify the solution to the following questions, with the answer given in the bracket at the end of each question:

A Tree Diagram as a Probabilistic Model

1. What is the probability of having at least 1 white ball? (10/11)
2. What is the probability that the balls are the same color? (17/33)
3. What is the probability that the second ball is white? (2/3)
4. What is the probability that the second ball is white given that the balls are the same color? (14/17)
5. Are the events in (2) and (3) independent? (No. Why Not?)
6. What is the expected number of white balls? (4/3)

Another Question for You: A coin fair is flipped twice, what is the conditional probability that both flips land on heads, given:

a.   The first flip lands on heads
b.   At least one of the flips lands on head.

Are the answers to part a and b identical? Why?

You may like using the Bayes' Revised Probability JavaScript.

Further Reading:
Ross Sh., A First Course in Probability, Prentice Hall, 2001.

---

Combinatorial Math: How to Count Without Counting

Many disciplines and sciences require the answer to the question: How Many? In finite probability theory we need to know how many outcomes there would be for a particular event, and we need to know the total number of outcomes in the sample space.

Combinatorics, also referred to as Combinatorial Mathematics, is the field of mathematics concerned with problems of selection, arrangement, and operation within a finite or discrete system. Its objective is: How to count without counting. Therefore, One of the basic problems of combinatorics is to determine the number of possible configurations of objects of a given type.

You may ask, why combinatorics? If a sample spaces contains a finite set of outcomes, determining the probability of an event often is a counting problem. But often the numbers are just too large to count in the 1, 2, 3, 4 ordinary ways.

A Fundamental Result: If an operation consists of two steps, of which the first can be done in n1ways and for each of these the second can be done in n2 ways, then the entire operation can be done in a total of n1× n2 ways.

This simple rule can be generalized as follow: If an operation consists of k steps, of which the first can be done in n1 ways and for each of these the second step can be done in n2 ways, for each of these the third step can be done in n3 ways and so forth, then the whole operation can be done in n1 × n2 × n3 × n4 ×.. × nk ways.

Numerical Example: A quality control inspector wishes to select one part for inspection from each of four different bins containing 4, 3, 5 and 4 parts respectively. The total number of ways that the parts can be selected is 4×3×5×4 or 240 ways.

Factorial Notation: the notation n! (read as, n factorial) means by definition the product:

n! = (n)(n-1)(n-2)(n-3)...(3)(2)(1).

Notice that by convention, 0! = 1, (i.e., 0! º 1) . For example, 6! = 6×5×4×3×2×1 = 720

Permutations versus Combination: A permutation is an arrangement of objects from a set of objects. That is, the objects are chosen from a particular set and listed in a particular order. A combination is a selection of objects from a set of objects, that is objects are chosen from a particular set and listed, but the order in which the objects are listed is immaterial.

Permutations Example: How many permutations (ordered arrangements) are there of the letters a,   b,  and  c? In this case it is easy to make a list:

abc,     bac,     cab
acb,     bca,     cba

The number of permutations is six. We might observe that there are 3 choices for the first letter, 2 choices for the second letter and 1 choice for the third letter. There is 3 × 2 × 1 = 3! permutations of the three letters a, b and c. Generalizing, if we have n distinct objects, we would have n choices for the first position, n-1 choices for the second position and so on. We find that the permutation of n objects selected among n distinct objects is n!.

The number of ways of lining up k objects at a time from n distinct objects is denoted by _n P ^k, and by the preceding we have:

_n P ^k = (n)(n-1)(n-2)(n-3)......(n-k+1)

Therfore, The number of permutations of n distinct objects taken k at a time can be written as:

_n P ^k = n! / (n - k) !

Combinations: There are many problems in which we are interested in determining the number of ways in which k objects can be selected from n distinct objects without regard to the order in which they are selected. Such selections are called combinations or k-sets. It may help to think of combinations as a committee. The key here is without regard for order.

The number of combinations of k objects from a set with n objects is _n C ^k. For example, the combinations of {1,2,3,4} taken k=2 at a time are {1,2}, {1,3}, {1,4}, {2,3}, {2,4}, {3,4}, for a total of 6 = 4! / [(2!)(4-2) !] subsets.

The general formula is:

_n C ^k = n! / [k! (n-k) !].

This is basically a subset problem where you specify the number of elements in the subset.

You may ask, what is the relation of combinations to permutations? Each of the above subsets forms 3! = 6 distinct permutations. 6×4 = 24, which equals 4P ³ If we use the notation ₄ C ³ to indicate the number of combinations of 4 distinct objects taken 3 at a time, then by the above we have:

₄ C ³ = ₄ P ³ / 3! = 4! / [ 3! (4 -3) !] = 4.

Notice that:

_n C ^k = _n C ^n-k

An Application: One of the fundamental aspects of economic activity is a trade in which one party provides another party something, in return for which the second party provides the first something else, i.e., the Barter Economics.

The invention of money during 16^th Century in Europe was a necessary tool of trading. The usage of money greatly simplifies barter system of trading, thus lowering transactions costs. If a society produces 100 different goods, there are:

₁₀₀ C ² = 100! / [2! (100 - 2)!] = 100(99)(98!) / [2 (98!)] = (100)(99)] / 2 = 4,950

different possible,"good-for-good" trades. With money, only 100 prices are needed to establish all possible trading ratios.

As another application, consider the following probabilistic problem. Suppose there are at most 10 defective items in a batch of size 150. You have shipped 15 items to one of your customers. What is the chance that the customer would find at least one defective item?

P [ at least one defective item] = 1 – P [no defective items] 1 – [ ( ₁₀ C ⁰ )( ₁₄₀ C ¹⁰ ) / ₁₅₀ C ¹⁵ ] = 2/3

This probability is too large, meaning it has a high risk to make an unsatisfied customer.

Permutation with Repetitions: How many different letter arrangements can be formed using the letters P E P P E R?

In general, there are multinomial coefficients:

n! / (n₁! n₂! n₃! ... n_r!)

different permutations of n objects, of which n₁ are alike, n₂, are alike, n₃ are alike,..... n_r are alike. Therefore, the answer is 6! /(3! 2! 1!) = 60 possible arrangements of the letters P E P P E R.

You may like using the Combinatorial Math JavaScript.

Further Reading:
Ross Sh., Introduction to Probability and Statistics for Engineers and Scientists, Academic Press, 2004.

---

Joint Probability and Statistics

A joint probability distribution of a group of random variables is the distribution of group of variables as a whole. Applied business statistics deal mostly with the joint probability distribution of two discrete random variables. The joint probability distribution of two discrete random variables is the likelihood of observing all combinations of the two variables.

Joint Probability Function: Let us have two discrete random variables X and Y, taking values x_i, i = 1,....,m, and y_i , j = 1,.....,n, respectively. The function:

P_{X, Y} = P_{X, Y}(x, y) = P(X = x, Y = y)

is called the joint probability function of the random variables X and Y.

As an example, consider two competitive stocks (A, and B). Suppose the estimated rates of return of stocks A and B are given as follow (respectively):

R_A = [0.8,   1.0,   1.2],   and   R_B = [0.9,   1.0,   1.1]

The numbers in the body of the following table are the estimated probabilities of all possible combinations of two jointly probabilities of the two random variables R_A, and R_B:

				RB
		0.9		1.0		1.1
	0.8	0.1		0.1		0.1
RA	1.0	0.1		0.1		0.1
	1.2	0.1		0.2		0.1

       Joint Probability

Marginal Probability Function: The function:

P_X(x) = åⁿ_j=1   P(X = x, Y = y_j)

is called the marginal density of X and similarly P_Y(y):

P_Y(y) = å^m_i=1   P(X = x_i, Y = y)

is called the marginal density of Y.

Numerical Example: Find the marginal density of R_A and R_B from the Joint Probability table.

To calculate the marginal distribution of R_B, simply look at the table and add the probabilities in each column.

To obtain the marginal distribution of R_A, add the probabilities in each row. The marginal distributions of A and B are shown at the rigt and the bottom margins of the below table, respectively:

						RB					Marginal
				0.9		1.0		1.1			¯
		0.8		0.1		0.1		0.1			0.3
RA		1.0		0.1		0.1		0.1			0.3
		1.2		0.1		0.2		0.1			0.4
Marginal		®		0.3		0.4		0.3

       Marginal Probability Functions

It is clear that a given joint distribution determines the marginal distributions uniquely. However, the converse is not true; a given marginal distribution can come from many different joint distributions. The function that links the marginal densities and the joint density is called the copula. In practice, one picks the marginal distributions first and then selects an appropriate copula to achieve the right amount of dependency among the individual random variables.

Cumulative Distribution: Take X and Y as above, then the function:

F_{X, Y} = F_{X, Y}(x, y) =   P(X £ x, Y £ y)

is called the joint cumulative distribution of X and Y.

				RB
		0.9		1.0		1.1
	0.8	0.1		0.2		0.3
RA	1.0	0.2		0.4		0.6
	1.2	0.3		0.7		1.0

Joint Cumulative Distribution

The resulting F must increase in the left-to-right and top-to-bottom directions.

The function:

F_X(x) = åⁿ_j=1   P(X £ x, Y = y_j)

is called the marginal cumulative distribution of X and similarly.

F_Y(y) = å^m_i=1   P(X = x _i, Y £ y)

is called the joint cumulative distribution of Y.

Stochastic Independence: When P(A | B) does not depend on the event B, that is P(A | B) = P(A) is given by:

P(A | B) = [P(A Ç B)] / [P(B)]     if P(B) > 0

and is left undefined when P(B) = 0. The symbol A Ç B means"the event A occurs and the event B occurs.

As an example, suppose we wish to compute the probability that the return on A is medium or high (R_A ³ 1.0) given that the return on B is medium or high (R_B ³ 1.0)?

We need to calculate:

P(R_A ³ 1.0  |  R_B ³ 1.0) = [P(R_A ³ 1.0  and  R_B ³ 1.0) ]  /  [P(R_B ³ 1.0)]

Now referring to the below tables:

				RB
		0.9		1.0		1.1
	0.8	0.1		0.1		0.1
RA	1.0	0.1		0.1		0.1
	1.2	0.1		0.2		0.1

P(R_A ³ 1.0 and R_B ³ 1.0 )

				RB
		0.9		1.0		1.1
	0.8	0.1		0.1		0.1
RA	1.0	0.1		0.1		0.1
	1.2	0.1		0.2		0.1

P(R_B ³ 1.0 )

P(R_A ³ 1.0 and R_B ³ 1.0) = 0.5, P(R_B ³ 1.0) = 0.7, and consequently:

P(R_A ³ 1 | R_B ³ 1) = [0.5 ] / [ 0.7] = 0.714.

An Application: Determine the number of elementary outcomes and then find the probability of the event 1/2(R_A + R_B) < 1.0.

Note that each return takes three values and is allowed to move independently of the other return, which means we have nine elementary outcomes. In the probabilities of the elementary outcomes that belong to the event 1/2(R_A + R_B) = 0.4 are given in bold:

				RB
		0.9		1.0		1.1
	0.8	0.1		0.1		0.1
RA	1.0	0.1		0.1		0.1
	1.2	0.1		0.2		0.1

Consequently,

P[1/2(R_A + R_B)] = 0.4.

For estimation of the expected values, variances, etc, you may using the Bivariate Distributions JavaScript.

Further Reading:
Ross Sh., Introduction to Probability Models, Academic Press, 2002.

---

Mutually Exclusive versus Independent Events

Mutually Exclusive (ME): Event A and B are ME if both cannot occur simultaneously. That is, P[A and B] = 0.

Independency (Ind.): Events A and B are independent if having the information that B already occurred does not change the probability that A will occur. That is P[A given B occurred] = P[A].

If two events are ME they are also Dependent: P(A given B) = P[A and B] ¸ P[B], and since P[A and B] = 0 (by ME), then P[A given B] = 0. Similarly,

If two events are Independent then they are also not ME.

If two events are Dependent then they may or may not be ME.

If two events are not ME, then they may or may not be Independent.

The following Figure contains all possibilities. The notations used in this table are as follows: X means does not imply, question mark ? means it may or may not imply, while the check mark means it implies.

Notice that the (probabilistic) pairwise independency and mutual independency for a collection of events A₁,..., A_n are two different notions.

Further Reading:
Ross Sh., A First Course in Probability, Prentice Hall, 2001.

---

What Is so Important About the Normal Distributions?

The term"normal" possibly arose because of the various attempts made to establish this distribution as the underlying rule governing all continuous variables. These attempts were based on false premises and consequently failed. Nonetheless, the normal distribution rightly occupies a preeminent place in the field of probability. In addition to portraying the distribution of many types of natural and physical phenomena (such as the heights of men, diameters of machined parts, etc.), it also serves as a convenient approximation of many other distributions which are less tractable. Most importantly, it describes the manner in which certain estimators of population characteristics vary from sample to sample and, thereby, serves as the foundation upon which much statistical inference from a random sample to population are made.

Normal Distribution (called also Gaussian) curves, which have a bell-shaped appearance (it is sometimes even referred to as the"bell-shaped curves") are very important in statistical analysis. In any normal distribution is observations are distributed symmetrically around the mean, 68% of all values under the curve lie within one standard deviation of the mean and 95% lie within two standard deviations.

There are many reasons for their popularity. The following are the most important reasons for its applicability:

1. One reason the normal distribution is important is that a wide variety of naturally occurring random variables such as heights and weights of all creatures are distributed evenly around a central value, average, or norm (hence, the name normal distribution). Although the distributions are only approximately normal, they are usually quite close.

   Whenever there are too many factors influencing the outcome of a random outcome, then the underlying distribution is approximately normal. For example, the height of a tree is determined by the"sum" of such factors as rain, soil quality, sunshine, disease, etc.

   As Francis Galton wrote in 1889, "Whenever a large sample of chaotic elements are taken in hand and arranged in the order of their magnitude, an unsuspected and most beautiful form of regularity proves to have been latent all along."

2. Almost all statistical tables are limited by the size of their parameters. However, when these parameters are large enough one may use normal distribution for calculating the critical values for these tables. For example, the F-statistic is related to standard normal z-statistic as follows: F = z², where F has (d.f.₁ = 1, and d.f.₂ is the largest available in the F-table). For more, visit the Relationships among Common Distributions.

   Approximation of the binomial: For example, the normal distribution provides a very accurate approximation of the binomial when n is large and p is close to 1/2. Even if n is small and p is not extremely close to 0 or to 1, the approximation is adequate. In fact, the normal approximation of the binomial will be satisfactory for most purposes provided that np > 5 and nq > 5.

   Here is how the approximation is made. First, set m = np and s² = npq. To allow for the fact that the binomial is a discrete distribution, we conventionally use a continuity correction factor of 1/2 unit added to or subtracted from X on the grounds that the discrete value (x = a) should correspond on a continuous scale to (a - 1/2) < x < (a + 1/2). Then we compute the value of the standard normal variable by:

   z = [(a - 1/2) - m]/s   OR    z = [(a + 1/2) - m]/s

   Now one may used the standard normal table for the numerical values.

   An Application: The probability of a defective item coming off a certain assembly line is p = 0.25. A sample of 400 items is selected from a large lot of these items. What is the probability 90 or less items are defective?

3. If the mean and standard deviation of a normal distribution are known, it is easy to convert back and forth from raw scores to percentiles.

4. It has been proven that the underlying distribution is normal if and only if the sample mean is independent of the sample variance, this characterizes the normal distribution. Therefore many effective transformations can be applied to convert almost any shaped distribution into a normal one.

5. The most important reason for popularity of normal distribution is the Central Limit Theorem (CLT). The distribution of the sample averages of a large number of independent random variables will be approximately normal regardless of the distributions of the individual random variables. The Central Limit Theorem is a useful tool when you are dealing with a population with an unknown distribution. Often, you may analyze the mean (or the sum) of a sample of size n. For example instead of analyzing the weights of individual items you may analyze the batch of size n, that is, the packages each containing n items.

6. The Sampling distribution of normal populations provide more information than any other distributions. For example, the following standard (i.e., having the same unit as the data have) errors are readily available:
   • Standard Error of the Median = (p/2n)^½S.

   • Standard Error of the Standard Deviation = S/(2n)^½.
     Therefore, the test statistic for the null hypothesis s = s₀, is Z = (2n)^½ (S - s₀)/s₀.

   • Standard Error of the Variance = S²[(2/(n-1)]^½.

   • Standard Error of the Interquartiles Half-Range (Q) = 1.166Q/n^½

   • Standard Error of the Skewness = (6/n)^½.

   • Standard Error of the Skewness of Sample Mean = Skewness/n^½

     Notice that the skewness in sampling distribution of the mean rapidly disappears as n gets larger.

   • Standard Error of the Kurtosis = (24/n)^½ = 2 times the standard error of skewness.

   • Standard Error of the Correlation (r) = [(1 - r²)/(n-1)]^½.

   Moreover,

   Quartile deviation » 2S/3,     and,    Mean absolute deviation » 4S/5.

7. The other reason the normal distributions are so important is that the normality condition is required by almost all kinds of parametric statistical tests. Using most statistical tables, such as T-table (except its last row), c²-table, and F-tables, all required the normality condition of the population. This condition must be tested before using these tables, otherwise the conclusion might be wrong.

---

What Is A Sampling Distribution?

A sampling distribution describes probabilities associated with a statistic when a random sample is drawn from the entire population.

The sampling distribution is the density (for a continuous statistic, such as an estimated mean), or probability function (for discrete statistic, such as an estimated proportion).

Derivation of the sampling distribution is the first step in calculating a confidence interval or carrying out a hypothesis testing for a parameter.

Example: Suppose that x1,.......,xn are a simple random sample from a normally distributed population with expected value m and known variance s². Then, the sample mean is normally distributed with expected value m and variance s²/n.

The main idea of statistical inference is to take a random sample from the entire particular population and then to use the information from the sample to make inferences about the particular population characteristics such as the mean m(measure of central tendency), the standard deviation (measure of dispersion, spread) s or the proportion of units in the population that have a certain characteristic. Sampling saves money, time, and effort. Additionally, a sample can provide, in some cases, as much or more accuracy than a corresponding study that would attempt to investigate an entire population. Careful collection of data from a sample will often provide better information than a less careful study that tries to look at everything.

Often, one must also study the behavior of the mean of sample values taken from different specified populations; e.g., for comparison purposes.

Because a sample examines only part of a population, the sample mean will not exactly equal the corresponding mean of the population m . Thus, an important consideration for those planning and interpreting sampling results is the degree to which sample estimates, such as the sample mean, will agree with the corresponding population characteristic.

In practice, only one sample is usually taken. In some cases a small"pilot sample" is used to test the data-gathering mechanisms and to get preliminary information for planning the main sampling scheme. However, for purposes of understanding the degree to which sample means will agree with the corresponding population mean m , it is useful to consider what would happen if 10, or 50, or 100 separate sampling studies, of the same type, were conducted. How consistent would the results be across these different studies? If we could see that the results from each of the samples would be nearly the same (and nearly correct!), then we would have confidence in the single sample that will actually be used. On the other hand, seeing that answers from the repeated samples were too variable for the needed accuracy would suggest that a different sampling plan (perhaps with a larger sample size) should be used.

A sampling distribution is used to describe the distribution of outcomes that one would observe from replication of a particular sampling plan.

Know that estimates computed from one sample will be different from estimates that would be computed from another sample.

Understand that estimates are expected to differ from the population characteristics (parameters) that we are trying to estimate, but that the properties of sampling distributions allow us to quantify, based on probability, how they will differ.

Understand that different statistics have different sampling distributions with distribution shape depending on (a) the specific statistic, (b) the sample size, and (c) the parent distribution.

Understand the relationship between sample size and the distribution of sample estimates.

Understand that increasing the sample size can reduce the variability in a sampling distribution.

See that in large samples, many sampling distributions can be approximated with a normal distribution.

Sampling Distribution of the Mean and the Variance for Normal Populations: Given the random variable X is distributed normally with mean m and standard deviation s, then for a random sample of size n:

• The sampling distribution of [ - m] ´ n^½ ¸ s, is the standard normal distribution.

• The sampling distribution of [ - m ] ´ n^½ ¸ S, is a T-distribution with parameter d.f. = n-1.

• The sampling distribution of [S²(n-1) ¸ s²], is a c²-distribution with parameter d.f. = n-1.

• For two independent samples, the sampling distribution of [S ₁² / S₂²], is an F-distribution with parameters d.f.₁ = n ₁-1, and d.f.₂= n ₂-1.

---

What Is The Central Limit Theorem?

The central limit theorem (CLT) is a "limit" that is "central" to statistical practice. For practical purposes, the main idea of the CLT is that the average (center of data) of a sample of observations drawn from some population is approximately distributed as a normal distribution if certain conditions are met. In theoretical statistics there are several versions of the central limit theorem depending on how these conditions are specified. These are concerned with the types of conditions made about the distribution of the parent population (population from which the sample is drawn) and the actual sampling procedure.

One of the simplest versions of the central limit theorem stated by many textbooks is: if we take a random sample of size (n) from the entire population, then, the sample mean which is a random variable defined by:

S x_i / n,

has a histogram which converges to a normal distribution shape if n is large enough. Equivalently, the sample mean distribution approaches to normal distribution as the sample size increases.

Some students having difficulty reconciling their own understanding of the central limit theorem with some of the textbooks statements. Some textbooks do not emphasize the on the independent, random samples of fixed-size n (say more than 30).

The shape of the sampling distributions for means - becomes increasingly normal as the sample size n becomes larger. The increasing sample size is what causes the distribution to become increasingly normal and the independence condition provides the Ön contraction of the standard deviation.

The CLT for proportion data, such as binary 0, 1, again the sampling distribution-- while becoming increasingly "bell-shaped" -- remains confined to the domain [0,1]. This domain represents a dramatic difference from a normal distribution, with has an unbounded domain. However, as n increases without bound, the "width" of the bell becomes very small so that the CLT "still works".

In applications of the central limit theorem to practical problems in statistical inference, however, we are more interested in how closely the approximate distribution of the sample mean follows a normal distribution for finite sample size, than in the limiting distribution itself. Sufficiently close agreement with a normal distribution allows us to use normal theory for making inferences about population parameters (such as the mean ) using the sample mean, irrespective of the actual form of the parent population.

It can be shown that, if the parent population has mean m and a finite standard deviation s, then the sample mean distribution has the same mean m but with smaller standard deviation which is s divided by n^½.

You know by now that, whatever the parent population is, the standardized variable Z = (X - m )/s will have a distribution with a mean m = 0 and standard deviation s =1 under random sampling. Moreover, if the parent population is normal, then Z is distributed exactly as the standard normal. The central limit theorem states the remarkable result that, even when the parent population is non-normal, the standardized variable is approximately normal if the sample size is large enough. It is generally not possible to state conditions under which the approximation given by the central limit theorem works and what sample sizes are needed before the approximation becomes good enough. As a general guideline, statisticians have used the prescription that, if the parent distribution is symmetric and relatively short-tailed, then the sample mean more closely approximates normality for smaller samples than if the parent population is skewed or long-tailed.

Under certain conditions, in large samples, the sampling distribution of the sample mean can be approximated by a normal distribution. The sample size needed for the approximation to be adequate depends strongly on the shape of the parent distribution. Symmetry (or lack thereof) is particularly important.

For a symmetric parent distribution, even if very different from the shape of a normal distribution, an adequate approximation can be obtained with small samples (e.g., 15 or more for the uniform distribution). For symmetric, short-tailed parent distributions, the sample mean more closely approximates normality for smaller sample sizes than if the parent population is skewed and long-tailed. In some extreme cases (e.g. binomial) sample sizes far exceeding the typical guidelines (e.g., over 30) are needed for an adequate approximation. For some distributions without first and second moments (e.g., one is known as the Cauchy distribution), the central limit theorem does not hold.

For some distributions, extremely large (impractical) samples would be required to approach a normal distribution. In manufacturing, for example, when defects occur at a rate of less than 100 parts per million, using, a Beta distribution yields an honest Confidence Interval (CI) of total defects in the population.

A question for you: Roll two perfectly balanced dice one time and the result will sum to an integer between 2 and 12. Which sum is most likely? (Hint: what CLT implies?)

---

An Illustration of CLT

Sampling Distribution of the Sample Means: Instead of working with individual scores, statisticians often work with means. What happens is that several samples are taken, the mean is computed for each sample, and then the means are used as the data, rather than individual scores being used. The sample is a sampling distribution of the sample means.

The central limit theorem explains why many distributions tend to be close to the normal distribution. The key ingredient is that the random variable being observed should be the sum or mean of many independent identically distributed random variables.

We can draw the probability distribution of the following random variables:

Sampling Distribution of Values (X): Consider the case where a single, fair die is rolled.

Here are the values that are possible and their probabilities.

X Values	1	2	3	4	5	6
Probability	1/6	1/6	1/6	1/6	1/6	1/6

Here are the mean, and variance of this random variable X:

Mean = m = E[X] = S [ x × p(x) ] = 3.5
Variance = s² = E[X²] – m² = S [ x² × p(x) ] - m² = 2.92

Sampling Distribution of Samples' Mean (Xbar): Consider the case where two fair dice are rolled instead of one.

Here are the sums that are possible and their probabilities.

Sum	2	3	4	5	6	7	8	9	10	11	12
Prob	1/36	2/36	3/36	4/36	5/36	6/36	5/36	4/36	3/36	2/36	1/36

But, we are not interested in the sum of the dice, we are interested in the sample mean. We find the sample mean by dividing the sum by the sample size.

Xbar	1.0	1.5	2.0	2.5	3.0	3.5	4.0	4.5	5.0	5.5	6.0
Prob	1/36	2/36	3/36	4/36	5/36	6/36	5/36	4/36	3/36	2/36	1/36

Now let us compute the mean, and variance, of this new random variable Xbar.

Here are the mean, and variance of the random variable Xbar:

Mean (Xbar) = m_Xbar = E[Xbar] = S [ xbar × p(xbar) ] = 3.5
Variance (Xbar> = s²_Xbar = E[Xbar²] – m_Xbar ² = S [ xbar² × p(xbar) ] - m_Xbar² = 1.46

Another way to think of sampling distributions is as Probability Distribution of Random Variables.

But, if we take repeated samples of the same size from a population, and then we plot the means of all those samples, our distribution will look a little better. We call distributions of sample statistics, Sampling Distributions.

The reason for this is that you can get the middle values in many more different ways than the extremes.

Example: When throwing two dice: 1+6 = 2+5 = 3+4 = 7, but only 1+1 = 2 and only 6+6 = 12. That is: even though you get any of the six numbers equally likely when throwing one die, the extremes are less probable than middle values in sums of several dice.

To see how the central limit theorem works is in the distribution of scores from increasing numbers of dice throws, as below.

In this illustration, the number on the top of each rolled die is an independent random event. Independent because the results of each die roll does not depend on the result of any previous roll, and random because, assuming that the die is "fair", the value on the top of the rolled die cannot be predicted in advance. The sum of their results is the total number of dots on the tops of all the rolled dice. The bar chart illustrates the distribution of the sum. The distribution of each independent die roll is flat, not bell-shaped. See for yourself. Roll one die a bunch of times and watch the bar chart evolve. The distribution of the sum of two independent die rolls is triangular. Try it and see. What about five dice? What about ten?

The CLT says that no matter what the distribution of the population looks like, the sampling distribution will be distributed normally, as long as your sample size is big enough (about 30). The distribution will have a mean equal to the population mean and a standard error equal to the population standard deviation divided by the square root of the sample size.

The measure of spread that we use for Sampling Distributions is the standard error (SE). The SE will always be smaller than the population standard deviation since the sampling distribution is one of sample statistics. Each sample mean will dampen the effect of outliers, bringing the tails of the sampling distribution in and creating a bigger "lump" in the middle, centered on the population mean. You can interpret the SE for sampling distributions in the same way as the standard deviation for populations.

Click on the image to enlarge it and THEN print it.
Limiting Behavior of the Sample Mean:
An Experimental Demonstration.

Properties of the Sampling Distribution of the Sample Means:

When all of the possible sample means are computed, then the following properties are true:

• The mean of the sample means will be the mean of the population
• The variance of the sample means will be the variance of the population divided by the sample size.
• The standard deviation of the sample means (known as the standard error of the mean) will be smaller than the population mean and will be equal to the standard deviation of the population divided by the square root of the sample size.
• If the population has a normal distribution, then the sample means will have a normal distribution.
• If the population is not normally distributed, but the sample size is sufficiently large, then the sample means will have an approximately normal distribution. Some books define sufficiently large as at least 30 and others as at least 31.

---

What Is"Degrees of Freedom"?

Recall that in estimating the population's variance, we used (n-1) rather than n, in the denominator. The factor (n-1) is called"degrees of freedom."

Estimation of the Population Variance: Variance in a population is defined as the average of squared deviations from the population mean. If we draw a random sample of n cases from a population where the mean is known, we can estimate the population variance in an intuitive way. We sum the deviations of scores from the population mean and divide this sum by n. This estimate is based on n independent pieces of information, and we have n degrees of freedom. Each of the n observations, including the last one, is unconstrained ('free' to vary).

When we do not know the population's mean, we can still estimate the population variance; but, now we compute deviations around the sample mean. This introduces an important constraint because the sum of the deviations around the sample mean is known to be zero. If we know the value for the first (n-1) deviations, the last one is known. There are only n-1 independent pieces of information in this estimate of variance.

If you study a system with n parameters x_i, i =1..., n, you can represent it in an n-dimension space. Any point of this space shall represent a potential state of your system. If your n parameters could vary independently, then your system would be fully described in a n-dimension hyper-volume (for n over 3). Now, imagine you have one constraint between the parameters (an equation with your n parameters), then your system would be described by a (n-1)-dimension hyper-surface (for n over 3). For example, in three dimensional space, a linear relationship means a plane which is 2-dimensional.

In statistics, your n parameters are your n data. To evaluate variance, you first need to infer the mean m . So when you evaluate the variance, you have one constraint on your system (which is the expression of the mean), and it remains only (n-1) degrees of freedom to your system.

Therefore, we divide the sum of squared deviations by n-1, rather than by n, when we have sample data. On average, deviations around the sample mean are smaller than deviations around the population mean. This is because our sample mean is always in the middle of our sample scores; in fact, the minimum possible sum of squared deviations for any sample of numbers is around the mean for that sample of numbers. Thus, if we sum the squared deviations from the sample mean and divide by n, we have an underestimate of the variance in the population (which is based on deviations around the population mean).

If we divide the sum of squared deviations by n-1 instead of n, our estimate is a bit larger, and it can be shown that this adjustment gives us an unbiased estimate of the population variance. However, for large n, say, over 30, it does not make too much difference if we divide by n, or n-1.

Degrees of Freedom in ANOVA: You will see the key parse"degrees of freedom" also appearing in the Analysis of Variance (ANOVA) tables. If I tell you about 4 numbers, but don't say what they are, the average could be anything. I have 4 degrees of freedom in the data set. If I tell you 3 of those numbers, and the average, you can guess the fourth number. The data set, given the average, has 3 degrees of freedom. If I tell you the average and the standard deviation of the numbers, I have given you 2 pieces of information, and reduced the degrees of freedom from 4 to 2. You only need to know 2 of the numbers' values to guess the other 2.

In an ANOVA table, degree of freedom (df) is the divisor in (Sum of Squared deviations)/df which will result in an unbiased estimate of the variance of a population.

In general, a degree of freedom d.f. = N - k, where N is the sample size, and k is a small number, equal to the number of"constraints", the number of"bits of information" already"used up". As we will see in the ANOVA section, degree of freedom is an additive quantity; total amounts of it can be "partitioned" into various components. For example, suppose we have a sample of size 13 and calculate its mean, and then the deviations from the mean; only 12 of the deviations are free to vary. Once one has found 12 of the deviations, the thirteenth one is determined.

In bivariate correlation or regression situations, k = 2. The calculation of the sample means of each variable"uses up" two bits of information, leaving N - 2 independent bits of information.

In a one-way analysis of variance (ANOVA) with g groups, there are three ways of using the data to estimate the population variance. If all the data are pooled, the conventional SST/(n-1) would provide an estimate of the population variance.

If the treatment groups are considered separately, the sample means can also be considered as estimates of the population mean, and thus SSb/(g - 1) can be used as an estimate. The remaining ("within-group","error") variance can be estimated from SSw/(n - g). This example demonstrates the partitioning of d.f.:
d.f. total = n - 1 = d.f.(between) + d.f.(within) = (g - 1) + (n - g).

Therefore, the simple 'working definition' of d.f. is ‘sample size minus the number of estimated parameters'. A more complete answer would have to explain why there are situations in which the degrees of freedom is not an integer. After we said all this, the best explanation, is mathematical in that we use d.f. to obtain an unbiased estimate.

In summary, the concept of degrees of freedom is used for the following two different purposes:

• Parameter(s) of certain distributions, such as F and t-distribution, are called degrees of freedom.

• Most importantly, the degrees of freedom are used to obtain unbiased estimates for the population parameters.

---

Applications of and Conditions for Using Statistical Tables

A problem with almost all statistical textbooks is that they not only do not provide information to understand connections between statistical tables. Students often ask: Why T- table values with d.f.=1 are much larger compared with other d.f. values? Some tables are limited, what should I do when the sample size is too large? How can I get familiarity with tables and their differences. Is there any type of integration among tables? Are there any connections between test of hypotheses and confidence interval under different scenario, for example testing with respect to one, two more than two populations? And so on.

The following Figure demonstrates useful relationships among common statistical tables:

Click on the image to enlarge it and THEN print it.
Relationships Among Common Statistical
Tables with Their Applications

Some widely used applications of the popular statistical tables can be categorized as follows:

T - Table:

1. Single Population µ Test.
2. Two Independent Populations Means Test.
3. The Before-and-After µ's Test.
4. Tests Concerning Regression Coefficients .
5. Test Concerning Correlation.

Conditions for using this table: Test for randomness of the data is needed before using this table. Test for normality condition of the population distribution is also needed if the sample size is small, or it may not be possible to invoke the central limit theorem.

Z - Table:

1. Test for Randomness.
2. Tests concerning µ for one population or two populations based on their large-size, random sample(s), (say over 30) to invoke the central limit theorem. This includes test concerning proportions, with large-size, random sample size n (say over 30) to invoke distribution convergence results.
3. To Compare Two Correlation Coefficients.

Notes: As you know by now, in test of hypotheses concerning m, and construction of confidence interval for it, we start with s known, since the critical value (and the p-value) of the Z-Table distribution can be used. Considering the more realistic situations, when we don't know s, the T-Table is used. In both cases, we need to verify the normality condition of the population's distribution; however, if the sample size n is very large, we can in fact switch back to Z-Table by virtue of the central limit theorem. For perfectly normal populations, the t-distribution corrects for any errors introduced by estimating s with s when doing inference.

Note also that, in hypothesis testing concerning the parameter of binomial and Poisson distributions for large sample sizes, the standard deviation is known under the null hypotheses. That's why you may use the normal approximations for both of these distributions.

Conditions for using this table: Test for randomness of the data is needed before using this table. Test for normality condition of the population distribution is also needed if the sample size is small, or it may not be possible to invoke the Central Limit Theorem.

Chi-square - Table:

1. Test for Cross-table Relationship.
2. Identical-Populations Test for Crosstable Data.
3. Test for Equality of Several Population Proportions.
4. Test for Equality of Several Population Medians.
5. Goodness-of-Fit Test for Probability Mass Functions.
6. Compatibility of Multi-Counts.
7. Correlation-Coefficient Testing.
8. Necessary Conditions in Applying the Above Tests.
9. Testing the Variance: Is the Quality that Good?.
10. Testing the Equality of Multi-Variances.

Conditions for using this table: The necessary conditions for using this table for all the above tests, except for the last one, can be found at Conditions for the Chi-square Based Tests. The last application requires normality (condition) of the population distribution.

F - Table:

1. Multi-Means Comparisons: Analysis of Variance (ANOVA).
2. Tests Concerning Two Variances.
3. Overall Assessment of Regression Models .

Conditions for using this table: Tests for randomness of the data and normality (condition) of the populations are needed before using this table for ANOVA. Same conditions must be satisfied for the residuals in regression analysis.

The following chart summarizes application of statistical tables with respect to test of hypotheses and construction of confidence intervals for mean mand variance s ² in one population or the comparison of two or more populations.

Click on the image to enlarge it and THEN print it.
Selection of an Appropriate Statistical Table

You may like using Online Statistical Computation in performing most of these tests. The P-values for the Popular Distributions Web site provides P-values useful in major statistical testing. The results are more accurate than those that can be obtained (by interpolation) from statistical tables of your textbook are.

Further Reading:
Balakrishnan N., and V. Nevzorov, A Primer on Statistical Distributions, Wiley, 2003.
Evans M., N. Hastings, and B. Peacock, Statistical Distributions, Wiley, 2000.
Kanji G., 100 Statistical Tests, Sage Publisher, 1995.

---

Numerical Examples for Statistical Tables

The presentation of the statistical tables is not universal. Some statistical textbooks authors’ enjoy given tabular values of the right-tail probabilities, while for others left-tail probabilities are preferred. Even within each of these groups you will find some differences in presenting each table differently than others, never in a unified format. This lack of uniformity often confuses most of students while learning statistics.

The following presents some numerical examples of common statistical tables with some applications. You may like using The P-values for the Popular Distributions JavaScript.

Binomial Probability

X ~ B(n, p), read, the random variable X has a binomial distribution with parameters n trials, and probability of a success is p.

Example: Find probability of at most k = 3 success from B(n = 7, p = 0.4). Using any Binomial table, one should get:
P[k £ 3] = 0.7102.
Using The P-values for the Popular Distributions JavaScript, one gets:

P[k £ 3] = 1 – P[k ³ 4] = 1 – 0.2898 = 0.7102.

Questions for you: Which of the following two events is more likely to happen? Getting exactly 6 heads in tossing a fair coin (i.e, p=1/2), n = 10 times or tossing it n=20 times. Why?

Application: A traveling salesman has find that the probability of a sale on a single contact is 0.02. If the salesman contacts 200 prospects, find the probability that he will make at least one sale.

P[at least one sale] = 1 – P[no sale] = 1 – (1-0.02)²⁰⁰ = 1 – (0.98)²⁰⁰ = 98%

Normal Density Function

X ~ N(0, 1), read, the random variable X is distributed Normally with mean, and variance 0, and 1, respectively.

A Fact: If X ~ N(m, s), then

Z = (X - m) / s ~ N(0, 1)

Example: Let X ~ N(1, 2), compute P(X £ 5.21)
P[ (X -1) / 2 £ (5.21 -1) / 2] = P(Z £ 2.105) » P(Z £ 2.11) = .4826 + .5 = .9826
Notice that P(Z £ 0) = .5

Similarly, P(X ³ 2.1) = P(Z ³ (2.1 - 1) / (2)) = P(Z ³ .55) = 0.5 - .2088 = .2912
Using The P-values for the Popular Distributions JavaScript, the 2p-value is:

P[| Z | £ 2.1] = 0.582.

Questions for you: Compute P( X ³ 3),  P(1 £ X £ 4),  P(X ³ - 1),   find the value of a such that P(X ³ a) = 0.4515

Applications:

1. Testing hypotheses on the population’s mean, with a known variance, at a given significance level a.

H₀: m = m₀     H_a: m ¹ m₀

A Fact: Given X ~ N(?, s) and having a random realization of size n: x₁, x ₂, ..., x_n, then

Z = [xbar_n - m] / (s / n^1/2)   ~ N(0,1).

Notice that in most cases, the standard deviation (s is unknown. However, one may use the sampling estimate S for s provided the sample size is large enough, say, over 30.)

Given n = 4, xbar₄ = 492  , test H₀: m = 500 at significance level a = 0.05 if s = 16?
The Z-statistic is Z = [492 - 500] / [16 / (4^1/2)] = -1, however the tabulated critical Z-value is Z_.025 = 1.96
Conclusion: No reason to reject H₀.

Question for you: Given the same sampling information, test H₀: m = 505 vs. H_a: m ¹ 505.

2. Setting a confidence interval on the mean, variance known.
Given xbar₄ = 492   construct a 95% confidence interval for m given s = 16

P [xbar - Z_a/2 s / n^1/2 £ m £ xbar + Z_a/2 s / n^1/2] ³ 1- a

Plugging in the numerical values, one gets:
P[476.3 £ m £ 507.7] ³ 0.95

Notice the Duality between the test of hypothesis and confidence interval.

Question for you:   Given the same sampling information, construct a 90% confidence interval for m given the same information.

3. Central Limit Theorem (CLT)

A Fact: If E(X) = m, Var(X) = s², then

(xbar - m) / (s / n^1/2) ~ N(0,1),

 for large n, say, n ³ 30

As a strong result, the CLT implies that if the sample size is large enough, then one may relax the normality condition whenever dealing with the question of testing or constructing confidence interval for population’s mean (m).

T-Density Function

A Fact: If X ~ N(m, ?), then

[xbar - m] / [S / n ^1/2] ~ t _n-1

Example: Find t such that P(T₁₁ > t) = .1 => t = 1.363
Using The P-values for the Popular Distributions JavaScript, the 2p-value is:

P[| T | £ 1.363] = 0.2.

Question for you: Find t such that P( T₈ > t ) = .01

Applications:

1. Testing hypotheses on mean, variance unknown
Given xbar₁₆ = 12.1 and S² = 2.225, test m = 12.5 vs m ¹ 12.5, at a = 0.05 significance level.
The computed statistic is t = -1.07 but the critical value from t-table = 2.131.
Conclusion: There is no reason to reject that m = 12.5.

Question for you: Given the same sampling information perform the test H₀: m = 11 vs m ¹ 11 at a = .01.

2. Construction of confidence interval for m, variance unknown
Example: Given xbar₁₆ = 12.1, S² = 2.225 develop a 95% confidence interval for m

P[xbar - t_{(a / 2, n -1)}   S / n^1/2 £ m £ xbar + t _{(a / 2, n- 1)}   S / n^1/2] ³ 1 - a

Therefore P[11.31 £ m £ 12.89] ³ 0.95 Again, notice the Duality between the test of hypothesis and confidence interval.

Question for you: Construct a 90% confidence interval for the same problem, is it wider than the other one, why or why not?

Notice that the T-density converges to the standard normal N(0, 1) as sample size gets larger. In fact the elements in the last row of t-table are the N(0,1) probabilities.

Chi-square Density Function

A Fact: If X ~ N(?, s), then the random variable

(n-1)S ² / s² ~ c ²_{(n - 1)}

the parameter (n-1) = n is called degrees-of-freedom (d.f).
Example: if d.f. = n = 15, and a = 0.975 find the c ² value. From the c ² table, we get c ² =6.26
Using The P-values for the Popular Distributions JavaScript, the p-value is:

P[c ² £ 6.26] = 0.975.

Applications:

1. Tests of hypotheses on the variance of a normal population.

Given n = 16 and S² = 2.22 test that s² = 2.0 at a = .05. The sampling statistic is c²₀ = 16.65, however from the table, the critical values are c²_{( 15, .025)} = 27.4884 and c²_{(15, .975)} = 6.26
Conclusion: There is no reason to reject that s² = 2.0

2. Interval estimation of the variance of a normal population

P[(n-1)S² / c² _{(n, a /2)} £ s² £ (n-1)S² / c²_{(n, 1-a /2)}] ³ 1 - a

Example: Given the same sampling information as above construct a 95% confidence interval for s
Plugging in the given information, you should get:
P[ 1.332 £ s² £ 4.587] ³ .95

Again, notice the Duality between the test of hypothesis and confidence interval.

Question for you: Given the same sampling information should we reject that s² = 2.0? at a = .1
Note that c²_{(15, .05)} = 8.55, and c²_{(15, .95)} = 7.26.

F-Density Function

A Fact: Consider two independent samples, one form two normal populations with known variance s²₁, and s₂², then

(S₁² / s₁²) / (S₁² / s₁²) ~ F_{(n 1 - 1, n2 - 1)}

Example: Find F such that P[F_{8, 7} ³ F] = .05 => The F value is F = 3.79

Notice: By now, you should have noticed that while every Statistical Table collected at the end of your textbook, provides the critical values for the right-tail as well as the left-tail probabilities, except the F-Table, which contains the critical values for the right-tail probabilities only. However, one might use the following nice property of F-distribution that:

F_{n1, n2, 1- a} = 1 / F_{n2, n1, a}

to obtain the critical values for the left-tail probabilities. Here is a numerical example:

F _{2, 3, 0.9} = 1 / F _{3, 2, 0.1} = 1 / 9.16 = 0.109

You need both tails probabilities for test of hypothesis and construction of confidence interval for the ratio of two independent populations' variances.

Example: Find P[F_{8, 7} ³ F] = .95. We may not be able to get the critical value from the table, however, one may utilize the fact that:

F_{n1, n2, 1 - a} = 1 / F_{n2, n1, a}

Therefore, F = 1 / 3.50 = 0.2857
Using The P-values for the Popular Distributions JavaScript, the p-value is: P[ F £ 0.2857] = 0.942 (which is exact).

Applications:

1. Testing of hypothesis on the variance of two normal populations.

Example: Given n₁ = n₂ = 16, S₁² = 34.14, and S₂² = 47.32, should we reject that s₁² = s₂² at a = 0.1
The sampling statistics is F = S₁² / S₂² = .785, but the critical values are F_{15, 15, .05} = 2.38, and F_{15, 15, .95} = 1 / 2.38 =0.421.
Conclusion: Therefore is no reason to reject.

Question for you: Given the same sampling information, construct a 90% confidence interval for variance ratio: s₁² / s₂²

---

Binomial Probability Function

An important class of decision problems under uncertainty involves situations for which there are only two possible random outcomes.

The binomial probability function gives probability of exact number of"successes" in n independent trials, when probability of success p on single trial is a constant. Each single trial is called a Bernoulli Trial satisfying the following conditions:

1. Each trial results in one of two possible, mutually exclusive, outcomes. One of the possible outcomes is denoted (arbitrarily) as a success, and the other is denoted a failure.
2. The probability of a success, denoted by p, remains constant from trial to trial. The probability of a failure, 1-p, is denoted by q.
3. The trials are independent; that is, the outcome of any particular trial is not affected by the outcome of any other trial.

The number of ways of getting r successes in n trials is:

P (r successes in n trials) = ⁿC_r . p^r . (1- p)^(n-r)
= n! / [r!(n-r)!] . [p^r . (1- p)^(n-r)].

The mean and variance of random variable r, are np and np(1-p), respectively, where q = 1 - p. The skewness and kurtosis are (2q -1)/ (npq)^½, and (1- 6pq)/(npq), respectively. From its skewness, we notice that the distribution is symmetric for p =1/2 and most skewed when p is 0 or 1.

Its mode is within interval [(n+1)p -1, (n+1)p], therefore if (n+1) p is not an integer, then the mode is an integer within the interval. However if (n+1)p is an integer, then its probability function has two but adjacent modes: (n+1)p -1, and (n+1)p.

Determination of probabilities for p over 0.5: The binomial tables in some textbooks are limited to deterring the probabilities for values of p up to 0.5. However, these tables can be used for values of p over 0.5. By recasting a problem in terms of p to 1 -p, and setting r to n-r, then the probability of obtaining r successes in n trials for a given value of p is equal to the probability of obtaining n-r failures in n trials with 1-p.

An Application: A large shipment of purchased parts is received at a warehouse, and a sample of 10 parts is checked for quality. The manufacturer's claim is that at most 5% might be defective. What is the chance that the sample includes one defective?

P (one defective out of ten) = {10! /[(1!)(9!)]}(0.05)¹(0.95)⁹ = 32%.

Know that the binomial distribution is to satisfy the five following requirements: (1) each trial can have only two outcomes or its outcomes can be reduced to two categories which are called pass and fail, (2) there must be a fixed number of trials, (3) the outcome of each trail must be independent, (4) the probabilities must remain constant, (5) and the outcome of interest is the number of successes.

Normal approximation for binomial: All binomial tables are limited in their scope; therefore it is necessary to use standard normal distribution in computing the binomial probabilities. The following numerical example illustrates how good the approximation could be. This provides an indication for real applications when n is beyond the given values in the available binomial tables.

Numerical Example: A sample of 20 items are taken randomly from a manufacturing process with defective probability p = 0.40. What is the probability of obtaining exactly 5 defective?

P (5 out of 20) = {20!/[(5!)(15!)]} ´ (0.40)⁵(0.6)¹⁵= 7.5%

Since the mean and standard deviation of distribution are:

m = np = 8, and s = (npq)^1/2 = 2.19,

respectively; therefore, the standardized observation for r = 5, by using the continuity factor (which always enlarges) are:

z₁ = [(r-1/2) - m] / s = (4.5 -8)/2.19 = -1.60, and

z₂ = [(r+1/2) - m] / s = (5.5 -8)/2.19 = -1.14.

Therefore, the approximated P (5 out of 20) is P (z being within interval -1.60, -1.14). Now, by using the standard normal table, we obtain:

P (5 out of 20) = 0.44520 - 0.37286 = 7.2%

Comments: The approximation for binomial distribution is used frequently in quality control, reliability, survey sampling, and other industrial problems.

Poisson approximation for binomial: Notice that, whenever you use Poisson approximation to the binomial distribution with parameters n and p, then the goodness of the approximation is largely determined by the smallness of the p parameter rather than how large is n.

You might like to use Common Discrete Probability Functions to obtain probability and the cumulative probability functions.

You might like to use the Exact Confidence Interval Construction and Test of Hypothesis for Binomial Population , and Binomial Probability Function JavaScript in performing some numerical experimentation for validating the above assertions for a deeper understanding.

---

Geometric Distribution

In a sequence of independent and identically distributed Bernoulli (p) trials, the number of trials required to get the 1st success has a Geometric(p) distribution.

A Typical Geometric Probability Function
Click on the image to enlarge it and THEN print it

If a single event or trial has two possible outcomes, say X_i can be 0 or 1 with P(X_i=1) = p, the probability of having to observe k trials before the first "one" appears is given by the geometric distribution.

The probability that the first "one" would appear on the first trial is p.
The probability that the first "one" appears on the second trial is p(1-p), because the first trial had to have been a zero followed by a one.
By generalizing this procedure, the probability that there will be k-1 failures before the first success is:

P (X = k) = (1 –p) ^k-1p

This is the geometric distribution.

A geometric distribution has a mean of 1/p and a variance of (1-p)/p².

Application: A manufacturing process is monitored. As each product exits the process line, it is tested for defective versus non-defective. On the first defect, the process is stopped for re-adjustment. The random variable X follows a Geometric distribution with p = P(product is non-defective).

The Geometric distribution has the memoryless property. Mathematically, for any non-negative integers s and t, this property can be written

P(X = s + t | X ³ s ) = P(X = t)

Application: Gives probability of requiring exactly x binomial trials before the first success is achieved. Used in quality control, reliability, and other industrial situations.

Example: Determination of probability of requiring exactly five tests firings before first success is achieved.

The Geometric distribution is the discrete analogue of the Exponential distribution, which models the time needed to get a success.

The Exponential distribution is the continuous analog of the Geometric distribution. Like the Geometric distribution, the Exponential distribution also has the memoryless property.

Mathematically, for any non-negative real numbers s and t, this property can be written

P(X > s + t | X > s ) = P(X > t)

The Exponential distribution is a special case of the Gamma distribution (r = 1). Furthermore, the sum of r independent and identically distributed Exponential (l) random variables has a Gamma distribution with parameters r and theta.

In a Poisson (l) process, the waiting times between consecutive events are distributed as Exponential with mean 1/(l).

You might like to use Common Discrete Probability Functions to obtain probability and the cumulative probability functions.

---

Negative Binomial Distribution

This is an extension of the geometric distribution, describing the waiting time until r "ones" have appeared. The probability of the r^th "one" appearing on the k^th trial is given by the negative binomial distribution:

P (X = k) = _r-1C ^k-1p^r-1 (1 –p) ^k-r p

in other words, the first part is the probability of r-1 success in the previous k-1 trails as a binomial probability, the last tem is the probability of success.

The following is a Negative Binomial probability function with parameters (r = 6 , k= 30, p = 0.5):

Click on the image to enlarge it and THEN print it.
A Negative Binomial Probability Function

A negative binomial distribution has:

mean = r/p and variance = r(1-p)/p²

Application: Suppose we are at a rifle range with an old gun that misfires 5 out of 6 times. Define ``success'' as the event the gunfires and let X be the number of failures before the third success. Then X has a negative binomial with parameters (3, 1/6). The probability that there are 10 failures before the third success is given by:

P(X = 10) = ₂C¹² (1/6)³ (5/6)¹⁰ = 5%

The expected value and variance of X are: E(X) = 3(1-5/6) / (1/6) = 15, and Var(X) = 3(1-5/6) / (1/6)² = 90.

In a sequence of independent and identically distributed Bernoulli (p) trials, the number of trials required to get the r^th success has a Negative Binomial (r,p) distribution.

Example: The number of oil wells that must be drilled to get r productive wells.

Relationships to Other Distributions: A Negative Binomial (r, p) random variable can be thought of as the sum of r independent and identically distributed Geometric(p) random variables. The Geometric (p) is a special case of the Negative Binomial with r=1.

Application: Gives probability similar to Poisson distribution when events do not occur at a constant rate and occurrence rate is a random variable that follows a gamma distribution.

Example: Distribution of number of cavities for a group of dental patients.

Comments: Generalization of Pascal distribution when s is not an integer. Many authors do not distinguish between Pascal and negative binomial distributions.

You might like to use Common Discrete Probability Functions to obtain probability and the cumulative probability functions.

---

Hypergeometric Distribution

The Hypergeometric (x; n, M, N) Distribution applies when we are sampling n items without replacement from a population of M successes and N-M failures.

The hypergeometric distribution arises when a random selection (without repetition) is made among objects of two distinct types. Typical examples:

Choose a team of 8 from a group of 10 men and 7 women.
Choose a committee of five from the legislature consisting of 52 Democrats and 48 Republicans.

The Concept of Hypergeometric Events

The above Venn diagram depicts choosing a random subset of size r from n items of which M = m items belong in a particular category, the probability that x = k of the selected items belong to that category.

The Binomial distribution looks at n trials "with replacement." The hypergeometric distribution is for the case "without replacement."

Here p changes from one Bernoulli trial to the next. Specifically, we have a population of size N with M out of the N members being "Successes" and the remaining (N-M) being "Failures." We choose a random sample of n (equivalent to taking out n members in succession without replacement).

The probability that X = x and given by:

P (X = x) = _xC ^M _n-xC ^N-M / _mC ^N

for all integers x between Max [0, n -(N+M)] and Min [n, M].

The expected value and variance of X are given by:

nM / N and nM(N-n)/(N-1)(N-M) / [N²(N-1)],

respectively.

In other words, there is a total number of N chips in the urn and n chips are drawn at random without replacement. Out of these n chips, k chips are red, and the remainder (n - k) are white. So, the formula is the number of ways to choose k chips from r red chips in the urn multiplied by the number of ways to choose n - k chips from white chips. This is divided by the sample space, or the number of ways to select n chips from the total of N chips in the urn.

Application: Gives probability of picking exactly x good units in a sample of n units from a population of N units when there are k bad units in the population. Used in quality control and related applications.

Example: Given a lot with 21 good units and four defective. What is the probability that a sample of five will yield not more than one defective?

Example: The number of defective items in a sample of size n from a box containing N items of which k are defective.

Application: A manufacturing process is monitored. As each product exits the process line, it is tested for defective versus non-defective. On the fifth defect, the process is stopped for re-adjustment. The random variable X follows a Negative Binomial distribution with r = 5 and p = P(product is non-defective).

Relationships to Other Distributions: The Hypergeometric (N, k, n) may be approximated by a Binomial (n, p = k/N) if N is very large relative to n. In this circumstance, replacement and non-replacement tend to become indistinguishable.

By extension, since the Binomial can be approximated by the Poisson, we can also approximate the Hypergeometric by a Poisson if the Binomial approximation is appropriate and n is reasonably large with k/N small.

You might like to use Common Discrete Probability Functions to obtain probability and the cumulative probability functions.

---

Exponential Density Function

An important class of decision problems under uncertainty concerns the random durations between events. For example, the the length of time between breakdowns of a machine not exceeding a certain time interval, such as the copying machine in your office not breaking down during this week.

Exponential distribution gives distribution of time between independent events occurring at a constant rate. Its density function is:

f(t) = l exp(-lt),

where l is the average number of events per unit of time, which is a positive number.

The mean and the variance of the random variable t (time between events) are 1/ l, and 1/l², respectively.

Applications include probabilistic assessment of the time between arrivals of patients to the emergency room of a hospital, and time between arrivals of ships at a particular port.

Comments: Itis a special case of Gamma distribution.

You might like to use Exponential Density to perform your computations, and Lilliefors Test for Exponentiality to perform the goodness-of-fit test.

---

F-Density Function

The F-distribution is the distribution of the ratio of two independent sampling (of size of n₁, and n₂, respectively) estimates of variance from standard normal distributions. It is also formed by the ratio of two independent chi-square variables divided by their respective independent degrees of freedom.

Its main applications are in testing equality of two independent population variances based on two independent random samples, ANOVA, and regression analysis.

By now, you should have noticed that while every Statistical Table collected at the end of your textbook, provides the critical values for the right-tail as well as the left-tail probabilities, except the F-Table, which contains the critical values for the right-tail probabilities only. However, one might use the following nice property of F-distribution that:

F_{n1, n2, 1- a} = 1 / F_{n2, n1, a}

to obtain the critical values for the left-tail probabilities. Here is a numerical example:

F _{2, 3, 0.9} = 1 / F _{3, 2, 0.1} = 1 / 9.16 = 0.109

You need both tails probabilities for test of hypothesis and construction of confidence interval for the ratio of two independent populations' variances.

You might like to use F-Density Function to obtain its P-values.

---

Chi-square Density Function

The probability density curve of a Chi-square distribution is an asymmetric curve stretching over the positive side of the line and having a long right tail. The form of the curve depends on the value of a parameter known as the degree of freedom (d.f.).

The expected value of Chi-square statistic is its d.f., its variance is twice of its d.f., and its mode is equal to (d.f.- 2).

Chi square Distribution relation to Normal Distribution: The Chi-square distribution is related to the sampling distribution of the variance when the sample is from a normal distribution. The sample variance is a sum of squares of standard normal variables N (0, 1). Hence, the of square of N (0,1) random variable is a Chi-square with 1 d.f..

Notice that the Chi-square is related to F-statistics as follows: F = Chi-square/d.f.₁, where F has (d.f.₁ = d.f. of the Chi-square-table, and d.f.₂ is the largest available in the F-table)

Similar to Normal random variables, the Chi-square has the additive property. For example, for two independent Chi-square variables, their sum is also Chi-square with degrees of freedom equal to the sum of the d.f. of the individual d.f.s. Thus the unbiased sample variance for a sample of size n from N (0,1) is a sum of n-1 Chi-squares, each with d.f. = 1, hence Chi-square with d.f. = n-1.

The most widely used applications of Chi-square distribution are:

The Chi-square Test for Association which is a non-parametric test; therefore, it can be used for nominal data too. It is a test of statistical significance widely used bivariate tabular association analysis. Typically, the hypothesis is whether or not two populations are different in some characteristic or aspect of their behavior based on two random samples. This test procedure is also known as the Pearson Chi-square test.

The Chi-square Goodness-of-Fit Test is used to test if an observed distribution conforms to any particular distribution. Calculation of this goodness-of-fit test is by comparison of observed data with data expected based on a particular distribution.

You might like to use Chi-square Density to find its P-values.

---

Multinomial Probability Function

A multinomial random variable is an extended binomial. However, the difference is that in a multinomial case, there are more than two possible outcomes. There are a fixed number of independent outcomes, with a given probability for each outcome.

The Expected Value (i.e., averages):

Expected Value = m = S (X_i ´ P_i),     the sum is over all i's.

Expected value is another name for the mean and (arithmetic) average.

It is an important statistic, because, your customers want to know what to “expect”, from your product/service OR as a purchaser of “raw material” for your product/service you need to know what you are buying, in other word what you expect to get:

To read-off the meaning of the above formula, consider computation of the average of the following data

2, 3, 2, 2, 0, 3

The average is Summing up all the numbers and dividing by their counts:

(2 + 3 + 2 + 2 + 0 + 3) / 6

This can be group and re-written as:

[ 2(3) + 3(2) + 0(1)] / 6 = 2(3/6) + 3(2/6) + 0(1/6)

which is the sum of each distinct observation times its probability. Right?

Expected value is known also as the First Moment, borrowed from Physics, because it is the point of balance where the data and the probabilities are the distances and the weights, respectively.

The Variance is:

Var(X) = E[(X- m)²] = E[X² - 2Xm + m²].

We simplify this using the above rules. First, because the expectation of a sum equals the sum of expectations:

Var(X) = E[X²] - E[2Xm] + E[m²].

Then, because constants may be taken out of an expectation:

Var(X) = E[X²] - 2 mE[X] + m² E[1] = E[X²] - 2 m ² + m2 = E[X²] - m².

Finally, notice that E[X²] can be written as E[g(X)] where g(X)=X². From the final fact about expectations, we can calculate this:

E[X²] = S x² P(X = x), for all x

Therefore, the Variance is:

Variance = s² = S [X_i² ´ P_i] - m²,     the sum is over all i's.

For example, suppose we toss two fair coins and we are interested in determining the expected value and the variance of the outcome:

E[X²] = (0) ²P(X=0) + (1) ²P(X=1) + (2) ²P(X=2) = 0(1/4) + 1(1/2) + 4(1/4) = 3/2.

From this, we calculate the variance:

Var(X) = E[X²] - m ² = 3/2 - (1) ² = 1/2.

Useful Tools for Population's Mean and Variance Estimations: It is not difficult to show that,

E(aX + b) = aE(X) + b, for any constant a and b
Var(aX+ b) = a²Var(X), for any constant a and b

Application: Notice that the above two examples are among some the tools well suited for reducing or even in preventing computational statistics round-off errors as well as computers' over/under flows.

Example: Suppose a random sample of size n = 9, is:

X:   220, 220, 260, 280, 270, 250, 300, 290, 240.

We wish to estimate the mean and the variance of the population based on this sample.

Let a = 10, and b = 22, then dividing the observational data set by a = 10, and then subtracting 22 fron each value, we obtain a new data set Y:

Y:   0, 0, 4, 6, 5, 3, 8, 7, 2.

Computing the mean and the variance of set Y, we obtain:

S y_i = 35, S y_i² = 203

Hence, the estimated mean and variance using the Y data set are 35/9, and [203 – 9(35/9)²] / 8 = 8.36, respectively. However, notice that X = 10Y + 22, therefore, the estimated mean and variance for the population are E(X) = 10 E(Y) + 22 = 350 + 22 = 372, and Var(X) = 10² Var(Y) = 836, respectively.

Notice that, the variance is not expressed in the same units as the expected value. So, the variance is hard to understand and to explain as a result of the squared term in its computation. This can be alleviated by working with the square root of the variance, which is called the Standard (i.e., having the same unit as the data have) Deviation:

Standard Deviation = s = (Variance) ^½

Both variance and standard deviation provide the same information and, therefore, one can always be obtained from the other. In other words, the process of computing standard deviation always involves computing the variance. Since standard deviation is the square root of the variance, it is always expressed in the same units as the expected value.

For the dynamic process, the Volatility as a measure for risk includes the time period over which the standard deviation is computed. The Volatility measure is defined as standard deviation divided by the square root of the time duration.

Coefficient of Variation: Coefficient of Variation (CV) is the absolute relative deviation with respect to size provided is not zero, expressed in percentage:

CV =100 |s/| %

Notice that the CV is independent from the expected value measurement. The coefficient of variation demonstrates the relationship between standard deviation and expected value, by expressing the risk as a percentage of the expected value. The inverse of CV (namely 1/CV) is called the Signal-to-Noise Ratio.

You might like to use Multinomial Applet for checking your computation and performing computer-assisted experimentation.

An Application: Consider two investment alternatives, Investment I and Investment II with the characteristics outlined in the following table:

- Two Investments -
Investment I			Investment II
Payoff %	Prob.		Payoff %	Prob.
1	0.25		3	0.33
7	0.50		5	0.33
12	0.25		8	0.34

Performance of Two Investments

To rank these two investments under the Standard Dominance Approach in Finance, first we must compute the mean and standard deviation and then analyze the results. Using the Multinomial for calculation, we notice that the Investment I has mean = 6.75% and standard deviation = 3.9%, while the second investment has mean = 5.36% and standard deviation = 2.06%. First observe that under the usual mean-variance analysis, these two investments cannot be ranked. This is because the first investment has the greater mean; it also has the greater standard deviation; therefore, the Standard Dominance Approach is not a useful tool here. We have to resort to the coefficient of variation (C.V.) as a systematic basis of comparison. The C.V. for Investment I is 57.74% and for Investment II is 38.43%. Therefore, Investment II has preference over the Investment I. Clearly, this approach can be used to rank any number of alternative investments. Notice that less variation in return on investment implies less risk.

Expectation of a sum of a random number of random variables: Suppose that the number of people entering a department store on a given day is a random variable with mean 50. Suppose further that the amount of money spent by these customers is independent random variables having a common mean of $80. What is the expected amount of money spent in the store on a given day?.

E (sum of N random variables X_i) = E(N) . E(X)

Hence, the expected amount of money spent in the store is (50)(80) = $4000.

You might like to use this JavaScript in performing some numerical experimentation to:

1. Show that E[aX + b] = aE(X) + b.
2. Show that V[aX + b] = a²V(X).
3. Show that: E(X²)= V(X) + (E(X))².

---

Normal Density Function

In the Descriptive Statistic Section of this Web site, we have been concerned with how empirical scores are distributed and how best to describe their distribution. We have discussed several different measures, but the mean m will be the measure that we use to describe the center of the distribution, and the standard deviation s will be the measure we use to describe the spread of the distribution. Knowing these two facts gives us ample information to make statements about the probability of observing a certain value within that distribution. If I know, for example, that the average Intelligence Quotient (I.Q.) score is 100 with a standard deviation of s = 20, then I know that someone with an I.Q. of 140 is very smart. I know this because 140 deviates from the mean mby twice the average amount as the rest of the scores in the distribution. Thus, it is unlikely to see a score as extreme as 140 because most of the I.Q. scores are clustered around 100 and only deviate 20 points from the mean m .

Many applications arise from the central limit theorem (CLT). The CLT states that, average of values of n observations approaches normal distribution, irrespective of the form of original distribution under quite general conditions. Consequently, normal distribution is an appropriate model for many, but not all, physical phenomena, such as distribution of physical measurements on living organisms, intelligence test scores, product dimensions, average temperatures, and so on.

Know that the Normal distribution is to satisfy seven requirements: (1) the graph should be bell shaped curve; (2) mean, median and mode are all equal; (3) mean, median and mode are located at the center of the distribution; (4) it has only one mode, (5) it is symmetric about mean, (6) it is a continuous function; (6) it never touches x-axis; and (7) the area under curve equals one.

Many methods of statistical analysis presume normal distribution.

When we know the mean and variance of a Normal then it allows us to find probabilities. So, if, for example, you knew some things about the average height of women in the nation, including the fact that heights are distributed normally, you could measure all the women in your extended family and find the average height. This enables you to determine a probability associated with your result, if the probability of getting your result, given your knowledge of women nationwide, is high. Then your family's female height cannot be said to be different from average. If that probability is low, then your result is rare (given the knowledge about women nationwide), and you can say your family is different. You have just completed a test of the hypothesis that the average height of women in your family is different from the overall average.

The ratio of two independent observations from the standard normal is distributed as the Cauchy Distribution which has thicker tails than a normal distribution. It density function is f(x) = 1/[p(1+x²)], for all real value x.

An Application: A portfolio manager believes that the overnight loss of his portfolio is distributed normally with mean $0 and standard deviation of $10 000. Find the 5% one-day value at risk for this portfolio.

Let X denotes the random portfolio loss distributed as X ~ N (0, 10 000²). The value at risk v_5% is defined by definition a number such that

P(X £ v_5%) = 0.95.

To find v_5% we standardize the random variable on the left-hand side:

X £ v_5%     Û     X – 0 £ v_5% – 0    Û    [X – 0] / [10 000] £ [v_5% – 0] / [10 000].

The transformation is denoted by Z = (X - 0) / 10 000 which has standard normal distribution. Therefore,

P{Z £ [v_5%– 0] / [10 000]} = 0.95.

If we denote by z_95% the 95% quantile of a standard normal distribution, then

[v_5%] / [10 000]   =   z_95%

v_5% can be found in normal statistical table:

z_95%  = 1.645, v_95%   = 10 000z_95%  = 16 450

Therefore, the overnight 5% value at risk is $16450.

You might like to use Standard Normal JavaScript instead of using tabular values from your textbook, and the well-known Lilliefors' Test for Normality to assess the goodness-of-fit.

---

Poisson Probability Function

Life is good for only two things, discovering mathematics and teaching mathematics.

-- Simeon Poisson                   

An important class of decision problems under uncertainty is characterized by the small chance of the occurrence of a particular event, such as an accident. Poisson probability function computes the probability of exactly x independent occurrences during a given period of time, if events take place independently and at a constant rate. Poisson probability function also represent number of occurrences over constant areas or volumes:

Poisson probabilities are often used; for example in quality control, software and hardware reliability, insurance claim, number of incoming telephone calls, and queuing theory.

Application: Gives probability of exactly x independent occurrences during a given period of time if events take place independently and at a constant rate. May also represents number of occurrences over constant areas or volumes. It is used frequently in quality control, reliability, queuing theory, and so on.

Example: Used to represent distribution of number of defects in a piece of material, customer arrivals, insurance claims, incoming telephone calls, alpha particles emitted, and so on.

A process that creates fabric is monitored. If the number of defects (X) per meter of fabric exceeds 5 then the process is stopped for diagnosis. The random variable X follows a Poisson distribution with rate = number of defects per meter of fabric.

An Application: One of the most useful applications of the Poisson distribution is in the field of queuing theory. In many situations where queues occur it has been shown that the number of people joining the queue in a given time period follows the Poisson model. For example, if the rate of arrivals to an emergency room is l per unit of time period (say 1 hr), then:

P ( n arrivals) = lⁿ  e^-l / n!

The mean and variance of random variable n are both l . However if the mean and variance of a random variable have equal numerical values, then it is not necessary that its distribution is a Poisson. Its mode is within interval [l -1, l].

Applications:

P ( 0 arrival) = e^-lP ( 1 arrival) = l  e^-l / 1!P ( 2 arrival) = l²  e^-l/ 2!

and so on. In general:

P ( n+1 arrivals ) = l P ( n arrivals ) / n.

Normal approximation for Poisson: All Poisson tables are limited in their scope; therefore, it is necessary to use standard normal distribution in computing the Poisson probabilities. The following numerical example illustrates how good the approximation could be.

Numerical Example: Emergency patients arrive at a large hospital at the rate of 0.033 per minute. What is the probability of exactly two arrivals during the next 30 minutes?

The arrival rate during 30 minutes is l = (30)(0.033) = 1. Therefore,

P (2 arrivals) = [1² /(2!)] e^-1 = 18%

The mean and standard deviation of distribution are:

m = l = 1, and s = l ^1/2 = 1,

respectively; therefore, the standardized observation for n = 2, by using the continuity factor (which always enlarges) are:

z₁ = [(r-1/2) - m] / s = (1.5 -1)/1 = 0.5, and

z₂ = [(r+1/2) - m] / s = (2.5 -1)/1 = 1.5.

Therefore, the approximated P (2 arrivals) is P (z being within the interval 0.5, 1.5). Now, by using the standard normal table, we obtain:

P (2 arrivals) = 0.43319 - 0.19146 = 24%

As you see the approximation is slightly overestimated, therefore the error is on the safe side. For large values of l, say over 20, one may use the Normal approximation to calculate Poisson probabilities.

Notice that by taking the square root of a Poisson random variable, the transformed variable is more symmetric. This is a useful transformation in regression analysis of Poisson observations.

Poisson approximation for binomial: Notice that, whenever you use Poisson approximation to the binomial distribution with parameters n and p, then the goodness of the approximation is largely determined by the smallness of the p parameter rather than how large is n.

You might like to use Common Discrete Probability Functions to obtain probability and the cumulative probability functions.

You might like to use Poisson Probability Function JavaScript to perform your computation, and Testing Poisson to perform the goodness-of-fit test.

Further Reading:
Barbour et al., Poisson Approximation, Oxford University Press, 1992.

---

Student T-Density Function

The t distributions were discovered in 1908 by William Gosset, who was a chemist and a statistician employed by the Guinness brewing company. He considered himself a student still learning statistics, so that is how he signed his papers as pseudonym"Student". Or, perhaps he used a pseudonym due to"trade secret" restrictions by Guinness.

Note that there are different t-distributions; it is a class of distributions. When we speak of a specific t distribution, we have to specify the degrees of freedom. The t density curves are symmetric and bell-shaped like the normal distribution and have their peak at 0. However, the spread is more than that of the standard normal distribution. The larger the degrees of freedom, the closer the t-density is to the normal density.

The shape of a t-distribution depends on a parameter called"degree-of-freedom". As the degree-of-freedom gets larger, the t-distribution gets closer and closer to the standard normal distribution. For practical purposes, the t-distribution is treated as the standard normal distribution when degree-of-freedom is greater than 30.

Suppose we have two independent random variables, one is Z, distributed as the standard normal distribution, while the other has a Chi-square distribution with (n-1) d.f.; then the random variable:

(n-1)Z / c²

has a t-distribution with (n-1) d.f. For large sample size (say, n over 30), the new random variable has an expected value equal to zero, and its variance is (n-1)/(n-3) which is close to one.

Notice that the t- statistic is related to F-statistic as follow: F = t², where F has (d.f.₁ = 1, and d.f.₂ = d.f. of the t-table)

You might like to use Student t-Density to obtain its P-values.

---

Triangular Density Function

The triangular distribution shows the number of successes when you know the minimum, maximum, and most likely values. For example, you could describe the number of intakes seen per week when past intake data show the minimum, maximum, and most likely number of cases seen. It has a continuous probability distribution.

The parameters for the triangular distribution are Minimum (a), Maximum (b), and Likeliest (c). There are three conditions underlying triangular distribution:

• The minimum number of items is fixed.
• The maximum number of items is fixed.
• The most likely number of items falls between the minimum and maximum values.

These three parameters forming a triangular shaped distribution, which shows that values near the minimum and maximum are less apt to occur than those near the most likely value.

The following are the general Triangular density function, together with the expected value and the variance for a Triangular random variable X (a, c, b):

f(x) = 2(x-a) / [(b-a)(c-a)], for a £ x £ c
f(x) = 2(b-x) / [(b-a)(b-a)], for c£ x £ b
E(X) = (a + b + c) / 3
Var(X) = (a² + b² + c² - ab - ac - bc) / 18

The following is a Triangular density function with parameters (a = 0, c = 0.25, a = 1):

Click on the image to enlarge it and THEN print it.
A Triangular Density Function

Application: Given X is distributed as above, compute the tails probability P (X £ 0.1 OR X ³ 0.9).

Further Reading:
Evans M., Hastings N., and B., Peacock, Triangular Distribution, Ch. 40 in Statistical Distributions, Wiley, pp. 187-188, 2000.

---

Uniform Density Function

The uniform density function gives the probability that observation will occur within a particular interval [a, b] when probability of occurrence within that interval is directly proportional to interval length. Its mean and variance are:

m = (a+b)/2,    s² = (b-a)²/12.

Applications: Used to generate random numbers in sampling and Monte Carlo simulation.

Comments: Special case of beta distribution.

You might like to use Goodness-of-Fit Test for Uniform and performing some numerical experimentation for a deeper understanding of the concepts.

Notice that any Uniform distribution has uncountable number of modes having equal density value; therefore it is considered as a homogeneous population.

Discrete Uniform Distribution: The discrete uniform distribution describes the distribution of n equally likely events (labeled with the integers from 1 to n), each with probability 1/n.

If X is a discrete uniform random variable with parameter n, then the mean, and variance are as follows:

E(X) = (n+1)/2, Var(X) = (n² -1) /12

Further Reading:
Balakrishnan N., and V. Nevzorov, A Primer on Statistical Distributions, Wiley, 2003.

---

Necessary Conditions for Statistical Decision Making

Introduction to Inferential Data Analysis Necessary Conditions: Do not just learn formulas and number-crunching. Learn about the conditions under which statistical testing procedures apply. The following conditions are common to almost all statistical tests:

1. Any undetected outliers may have major impact and may influence the results of almost all statistical estimation and testing procedures.

2. Homogeneous population. That is, there is not more than one mode. Perform Test for Homogeneity of a Population

3. The sample must be random. Perform Test for Randomness.

4. In addition to the Homogeneity requirement, each population has a normal distribution. Perform the Lilliefors' Test for Normality.

5. Homogeneity of variances. Variation in each population is almost the same as in the other(s). Perform The Bartlett's Test.

   For two populations use the F-test. For 3 or more populations, there is a practical rule known as the"Rule of 2". In this rule, one divides the highest variance of a sample by the lowest variance of the other sample. Given that the sample sizes are almost the same, and the value of this division is less than 2, then the variations of the populations are almost the same.

   Notice: This important condition in analysis of variance (ANOVA and the t-test for mean differences) is commonly tested by the Levene test or its modified test known as the Brown-Forsythe test. Interestingly, both tests rely on the homogeneity of variances condition!

These conditions are crucial, not for the method of computation, but for the testing using the resultant statistic. Otherwise, we can do ANOVA and regression without any assumptions, and the numbers come out the same. Simple computations give us least-square fits, partitions of variance, regression coefficients, and so on. We do need the above conditions when test of hypotheses are our main concern.

Further Readings:
Good Ph., and J. Hardin, Common Errors in Statistics, Wiley, 2003.
Wang H., Improved confidence estimators for the usual one-sided confidence intervals for the ratio of two normal variances, Statistics & Probability Letters, Vol. 59, No.3, 307-315, 2002.

---

Measure of Surprise for Outlier Detection

Robust statistical techniques are needed to cope with any undetected outliers; otherwise they are more likely to invalidate the conditions underlying statistical techniques, and they may seriously distort estimates and produce misleading conclusions in test of hypotheses. A common approach consists of assuming that contaminating models, different from the one generating the rest of the data, generate the (possible) outliers.

Because of a potentially large variance, outliers could be the outcome of sampling errors or clerical errors such as recording data. Therefore, you must be very careful and cautious. Before declaring an observation"an outlier," find out why and how such observation occurred. It could even be an error at the data entering stage while using any computer package.

In practice, any observation with a standardized value greater than 2.5 in absolute value is a candidate for being an outlier. In such a case, one must first investigate the source of the datum. If there is no doubt about the accuracy or veracity of the observation, then it should be removed, and the model should be refitted.

1. Compute the mean () and standard deviation (S) of the whole sample.

2. Set limits for the mean : - k ´ S,       + k ´ S. A typical value for k is 2.5

3. Remove all sample values outside the limits.

4. Now, iterate through the algorithm, the sample set may reduce after removing the outliers by applying step 3.

5. In most cases, we need to iterate through this algorithm several times until all outliers are removed.

An Application: Suppose you ask ten of your classmates to measure a given length X. The results (in mm) are:

46, 48, 38, 45, 47, 58, 44, 45, 43, 44

Is 58 an outlier? Computing the mean and the variance of the ten measurement using the Descriptive Sampling Statistics JavaScript, are 45.8, and 5.1(after the needed adjustment), respectively. The Z-value for 58 is Z (58) = 2.4. Since the measurements, in general, follow a normal distribution, therefore,

Probability [X as large as 2.4 times standard deviation] = 0.008,

obtained by using the Standard Normal P-value JavaScript, or from the normal table in your textbook.

According this probability, one expects only 0.09 of the ten measurements as bad as this one. This is a very rare event, however, in spite of such small probability, it has occurred, therefore, it might be an outlier.

The next most suspected measurement is 38, is it an outlier? It is a question for you.

A Notice: Outlier detection in the single population setting is not too difficult. Quite often, however, one can argue that the detected outliers are not really outliers, but form a second population. If this is the case, a data separation approach needs to be taken.

You might like to use the Identification of Outliers JavaScript in performing some numerical experimentation for validating and for a deeper understanding of the concepts

Further Reading:
Rothamsted V., V. Barnett, and T. Lewis, Outliers in Statistical Data, Wiley, 1994.

---

Homogeneous Population

A homogeneous population is a statistical population which has a unique mode.

Notice that, e.g., a Uniform distribution has uncountable number of modes having equal density value; therefore it is considered as a homogeneous population.

To determine if a given population is homogeneous or not, construct the histogram of a random sample from the entire population. If there is more than one mode, then you have a mixture of two or more different populations. Know that to perform any statistical testing, you need to make sure you are dealing with a homogeneous population.

One of the main applications of histogramming is to Test for Homogeneity of a Population. The unimodality of the histogram is a necessary condition for the homogeneity of a population in order to conduct any meaningful statistical analysis. However, notice that, e.g., a Uniform distribution has uncountable number of modes having equal density value; therefore it is considered as a homogeneous population.

---

Test for Randomness: The Runs' Test

A basic condition in almost all inferential statistics is that a set of data constitutes a random sample from a given homogeneous population. The condition of randomness is essential to make sure the sample is truly representitive of the population. The widely used test for randomness is the Runs test.

A"run" is a maximal subsequence of like elements.

Consider the following sequence (D for Defective items, N for Non-defective items) from a production line: DDDNNDNDNDDD. Number of runs is R = 7, with n₁ = 8, and n₂ = 4 which are number of D's and N's.

A sequence is a random sequence if it is neither"over-mixed" nor"under-mixed". An example of over-mixed sequence is DDDNDNDNDNDD, with R = 9 while under-mixed looks like DDDDDDDDNNNN with R = 2. There the above sequence seems to be a random sequence.

The Runs Tests, which is also known as Wald-Wolfowitz Test, is designed to test the randomness of a given sample at 100(1- a)% confidence level. To conduct a runs test on a sample, perform the following steps:

Step 1: compute the mean of the sample.

Step 2: going through the sample sequence, replace any observation with +, or - depending on whether it is above or below the mean. Discard any ties.

Step 3: compute R, n₁, and n₂.

Step 4: compute the expected mean and variance of R, as follows:

a =1 + 2n₁n₂/(n ₁ + n₂).

s² = 2n₁n₂(2n ₁n₂-n₁- n₂)/[[n₁ + n₂)² (n₁ + n₂ -1)].

Step 5: Compute z = (R-m)/ s.

Step 6: Conclusion:

If z > Z_a, then there might be cyclic, seasonality behavior (under-mixing).

If z < - Z_a, then there might be a trend.

If z < - Z_a/2, or z > Z_a/2, reject the randomness.

Note: This test is valid for cases for which both n₁ and n₂ are large, say greater than 10. For small sample sizes, special tables must be used.

For example, suppose for a given sample of size 50, we have R = 24, n₁ = 14 and n₂ = 36. Test for randomness at a = 0.05.
The Plugging these into the above formulas we have a = 16.95, s = 2.473, and z = -2.0 From Z-table, we have Z = 1.645. Therefore, there might be a trend, which means that the sample is not random.

You may use the following JavaScript to Test for Randomness.

---

Test for Normality

The standard test for normality is the Lilliefors' statistic. A histogram and normal probability plot will also help you distinguish between a systematic departure from normality when it shows up as a curve.

Lilliefors' Test for Normality: This test is a special case of the Kolmogorov-Smirnov goodness-of-fit test, developed for testing the normality of population's distribution. When applying the Lilliefors test, a comparison is made between the standard normal cumulative distribution function, and a sample cumulative distribution function with standardized random variable. If there is a close agreement between the two cumulative distributions, the hypothesis that the sample was drawn from population with a normal distribution function is supported. If, however, there is a discrepancy between the two cumulative distribution functions too great to be attributed to chance alone, then the hypothesis is rejected.

The difference between the two cumulative distribution functions is measured by the statistic D, which is the greatest vertical distance between the two functions.

You might like to use the well-known Lilliefors' Test for Normality to assess the goodness-of-fit.

Further Readings
Thode T., Testing for Normality, Marcel Dekker, Inc., 2001. Contains the major tests for normality.

---

Introduction to Estimation

To estimate means to esteem (to give value to). An estimator is any quantity calculated from the sample data which is used to give information about an unknown quantity in the population. For example, the sample mean is an estimator of the population mean m.

Results of estimation can be expressed as a single value; known as a point estimate, or a range of values, referred to as a confidence interval. Whenever we use point estimation, we calculate the margin of error associated with that point estimation.

Estimators of population parameters are sometimes distinguished from the true value by using the symbol 'hat'. For example, true population standard deviation s is estimated from a sample population standard deviation.

Again, the usual estimator of the population mean is = Sx_i / n, where n is the size of the sample and x₁, x₂, x₃,.......,x_n are the values of the sample. If the value of the estimator in a particular sample is found to be 5, then 5 is the estimate of the population mean µ.

---

Qualities of a Good Estimator

A"Good" estimator is the one which provides an estimate with the following qualities:

Unbiasedness: An estimate is said to be an unbiased estimate of a given parameter when the expected value of that estimator can be shown to be equal to the parameter being estimated. For example, the mean of a sample is an unbiased estimate of the mean of the population from which the sample was drawn. Unbiasedness is a good quality for an estimate, since, in such a case, using weighted average of several estimates provides a better estimate than each one of those estimates. Therefore, unbiasedness allows us to upgrade our estimates. For example, if your estimates of the population mean µ are say, 10, and 11.2 from two independent samples of sizes 20, and 30 respectively, then a better estimate of the population mean µ based on both samples is [20 (10) + 30 (11.2)] (20 + 30) = 10.75.

Consistency: The standard deviation of an estimate is called the standard error of that estimate. The larger the standard error the more error in your estimate. The standard deviation of an estimate is a commonly used index of the error entailed in estimating a population parameter based on the information in a random sample of size n from the entire population.

An estimator is said to be"consistent" if increasing the sample size produces an estimate with smaller standard error. Therefore, your estimate is"consistent" with the sample size. That is, spending more money to obtain a larger sample produces a better estimate.

Efficiency: An efficient estimate is one which has the smallest standard error among all unbiased estimators.

The"best" estimator is the one which is the closest to the population parameter being estimated:

Click on the image to enlarge it and THEN print it.
The Concept of "Distance" for an Estimator Is Demonstrated

The above figure illustrates the concept of closeness by means of aiming at the center for unbiased with minimum variance. Each dart board has several samples:

The first one has all its shots clustered tightly together, but none of them hit the center. The second one has a large spread, but around the center. The third one is worse than the first two. Only the last one has a tight cluster around the center, therefore has good efficiency.

If an estimator is unbiased, then its variability will determine its reliability. If an estimator is extremely variable, then the estimates it produces may not on average be as close to the population parameter as a biased estimator with small variance.

The following chart depicts the quality of a few popular estimators for the population mean µ:

Click on the image to enlarge it and THEN print it.
Sample Mean as a "Good" Estimator for the Population's Expected Value

The widely used estimator of the population mean m is = Sx_i/n, where n is the size of the sample and x₁, x₂, x₃,......., x_n are the values of the sample that have all of the above good properties. Therefore, it is a"good" estimator.

If you want an estimate of central tendency as a parameter for a test or for comparison, then small sample sizes are unlikely to yield any stable estimate. The mean is sensible in a symmetrical distribution as a measure of central tendency; but, e.g., with ten cases, you will not be able to judge whether you have a symmetrical distribution. However, the mean estimate is useful if you are trying to estimate the population sum, or some other function of the expected value of the distribution. Would the median be a better measure? In some distributions (e.g., shirt size) the mode may be better. BoxPlot will indicate outliers in the data set. If there are outliers, the median is better than the mean as a measure of central tendency.

You might like to use Descriptive Statistics JavaScript for obtaining"good" estimates.

Further Readings
Casella G., and R. Berger, Statistical Inference, Wadsworth Pub. Co., 2001.
Lehmann E., and G. Casella, Theory of Point Estimation, Springer Verlag, New York, 1998.

---

Estimations with Confidence

In practice, a confidence interval is used to express the uncertainty in a quantity being estimated. There is uncertainty because inferences are based on a random sample of finite size from the entire population or process of interest. To judge the statistical procedure we can ask what would happen if we were to repeat the same study, over and over, getting different data (and thus different confidence intervals) each time.

In most studies, investigators are usually interested in determining the size of difference of a measured outcome between groups, rather than a simple indication of whether or not it is statistically significant. Confidence intervals present a range of values, on the basis of the sample data, in which the value of such a difference may lie.

Know that a confidence interval computed from one sample will be different from a confidence interval computed from another sample.

Understand the relationship between sample size and width of confidence interval, moreover, know that sometimes the computed confidence interval does not contain the true value.

Let's say you compute a 95% confidence interval for a mean m . The way to interpret this is to imagine an infinite number of samples from the same population, at leat 95% of the computed intervals will contain the population mean m , and at most 5% will not. However, it is wrong to state,"I am 95% confident that the population mean m falls within the interval."

Again, the usual definition of a 95% confidence interval is an interval constructed by a process such that the interval will contain the true value at least 95% of the time. This means that"95%" is a property of the process, not the interval.

Is the probability of occurrence of the population mean greater in the confidence interval (CI) center and lowest at the boundaries? Does the probability of occurrence of the population mean in a confidence interval vary in a measurable way from the center to the boundaries? In a general sense, normality condition is assumed, and then the interval between CI limits is represented by a bell shaped t distribution. The expectation (E) of another value is highest at the calculated mean value, and decreases as the values approach the CI limits.

Tolerance Interval and CI: A good approximation for the single measurement tolerance interval is n^½ times confidence interval of the mean.

Statistics with Confidence
Click on the image to enlarge it and THEN print it

You may use Estimations With Confidence, and Confidence Intervals for Two Populations to check your hand computations.

You need to use Sample Size Determination JavaScript at the design stage of your statistical investigation in decision making with specific subjective requirements.

A Note on Multiple Comparison via Individual Intervals: Notice that, if the confidence intervals from two samples do not overlap, there is a statistically significant difference, say at 5%. However, the other way is not true; two confidence intervals can overlap even when there is a significant difference between them.

As a numerical example, consider the means of two independent samples. Suppose their values are 10 and 22 with equal standard error of 4. The 95% confidence interval for the two statistics (using the critical value of 1.96) are: [2.2, 17.8] and [14.2, 29.8], respectively. As you see they display considerable overlap. However, the z-statistic for the two-population mean is: |22 -10|/(16 + 16)^½ = 2.12 which is clearly significant under the same conditions as applied for constructing the confidence intervals.

One should examine the confidence interval for the difference explicitly. Even if the confidence intervals are overlapping, it is hard to find the exact overall confidence level. However, the sum of individual confidence levels can serve as an upper limit. This is evident from the fact that: P(A and B) £ P(A) + P(B).

Numerical examples for construction of confidence intervals are given in The Statistical Tables section.

Further Reading:
Cohen J., Statistical Power Analysis for the Behavioral Sciences, L. Erlbaum Associates, 1988.
Kraemer H., and S. Thiemann, How Many Subjects? Provides basic sample size tables, explanations, and power analysis.
Murphy K., and B. Myors, Statistical Power Analysis, L. Erlbaum Associates, 1998. Provides a simple and general sample size determination for hypothesis tests.
Newcombe R., Interval estimation for the difference between independent proportions: Comparison of eleven methods, Statistics in Medicine, 17, 873-890, 1998.
Hahn G. and W. Meeker, Statistical Intervals: A Guide for Practitioners, Wiley, 1991.
Schenker N., and J. Gentleman, On judging the significance of differences by examining the overlap between confidence intervals, The American Statistician, 55(2), 135-139, 2001.

---

What Is the Margin of Error?

Estimation is the process by which sample data are used to indicate the value of an unknown quantity in a population.

Results of estimation can be expressed as a single value, known as a point estimate; or a range of values, referred to as a confidence interval.

Whenever we use point estimation, we calculate the margin of error associated with that point estimate. For example, for the estimation of the population proportion, by the means of sample proportion (p), the margin of error is calculated often as follows:

±1.96 [p(1-p)/n]^1/2

In newspapers and television reports on public opinion polls, the margin of error often appears in a small font at the bottom of a table or screen. However, reporting the amount of error only, is not informative enough by itself, what is missing is the degree of the confidence in the findings. The more important missing piece of information is the sample size n; that is, how many people participated in the survey, 100 or 100000? By now, you know well that the larger the sample size the more accurate is the finding, right?

The reported margin of error is the margin of"sampling error". There are many non-sampling errors that can and do affect the accuracy of polls. Here we talk about sampling error. The fact that sub-groups might have sampling error larger than the group, one must include the following statement in the report:

"Other sources of error include, but are not limited to, individuals refusing to participate in the interview and inability to connect with the selected number. Every feasible effort was made to obtain a response and reduce the error, but the reader (or the viewer) should be aware that some error is inherent in all research."

If you have a yes/no question in a survey, you probably want to calculate a proportion P of Yes's (or No's). In a simple random sample survey, the variance of p is p(1-p)/n, ignoring the finite population correction, for large n, say over 30. Now a 95% confidence interval is

p - 1.96 [p(1-p)/n]^1/2,   p + 1.96 [p(1-p)/n]^1/2.

A conservative interval can be calculated, since p(1-p) takes its maximum value when p = 1/2. Replace 1.96 by 2, put p = 1/2 and you have a 95% consevative confidence interval of 1/n^1/2. This approximation works well as long as p is not too close to 0 or 1. This useful approximation allows you to calculate approximate 95% confidence intervals.

For continuous random variables, such as the estimation of the population mean m, the margin of error is calculated often as follows:

±1.96 S/n^1/2.

The margin of error can be reduced by one or a combination of the following strategies: