Independence

in probability theory

One of the most important notions in probability theory. Other terms occasionally used are statistical independence, stochastic independence. The assumption that the events, trials and random variables being considered are independent has long been a common premise, from the very beginnings of mathematical probability theory.

Independence of two random events is defined as follows. Let and be two random events, and let and be their probabilities. The conditional probability of given that has occurred is defined by

where is the probability of the joint occurrence of and . The events and are said to be independent if

(1)

If this is equivalent to

(2)

The meaning of this definition can be explained as follows. On the assumption that a large number of trials is being carried out, and assuming for the moment that (2) refers to relative frequencies rather than probabilities, one may conclude that the relative frequency of the event in all trials must be equal to the relative frequency of its occurrences in the trials in which also occurs. Thus, independence of two events indicates that there is no discernable connection between the occurrence of the one event and that of the other. Thus, the event "a randomly-selected person has a family name beginning, say, with the letter A" , and the event that "the same person will win the grand prize in the next play of the state lottery" are independent.

The definition of independence of random events , , may be presented in several equivalent versions. According to one version, these events are said to be independent if, for any , , and for any pairwise distinct natural numbers , the probability of the joint occurrence of the events is equal to the product of their probabilities:

(3)

Hence, as before, one may conclude that the conditional probability of each event given the occurrence of any combination of the others is equal to its "unconditional" probability.

Sometimes, besides the independence (mutual independence) of the events , one also considers the notion known as pairwise independence: Any two of these events, say and , , are independent. Independence of events implies pairwise independence, but the converse need not be true.

Prior to the axiomatic construction of probability theory, the independence concept was not interpreted in an adequately clear-cut fashion. In the words of A.A. Markov ([1], p. 24): "The concept of independent events may be considered quite clear in known theoretical problems; in other problems, however, the concept may of course become quite obscured, in keeping with the obscurity of the fundamental notion of probability" .

In the context of an axiomatic approach, the most natural definition of independence is the following. Let be some probability space, where is the set of elementary events, a -algebra of events and a probability measure defined on . One first defines independence of classes of events (the only classes considered here will be sub -algebras of ). Classes are said to be independent (relative to ) if any events are independent in the sense of (3); the classes (, where is an arbitrary index set) are said to be independent if, for any integer and any pairwise distinct , the classes are independent. Independence of events , , is equivalent to independence of the classes

In the case of trials, independence is precisely the independence of the -algebras generated by the trials.

For random variables , , independence is defined as independence of the sub -algebras , where is the pre-image under of the -algebra of Borel sets on the real line. Independence of random events is equivalent to independence of their indicators , i.e. independence of the random variables defined by

and

There are various necessary and sufficient conditions for the independence of random variables :

1) For arbitrary real numbers , the value of the distribution function

is equal to the product of the values of the individual distribution functions

2) If there exist densities (cf. Density of a probability distribution), then the density is equal to the product of the individual densities for almost all with respect to Lebesgue measure on .

3) The characteristic function

is equal, for all real numbers , to the product , , of the individual characteristic functions.

The most important schemes of probability theory are based on the assumption that various events and random variables are independent: sequences of independent random variables (see, e.g., Bernoulli random walk; Law of large numbers; Limit theorems of probability theory), stochastic processes with independent increments (see, e.g., Wiener process; Stochastic process), etc. (see also Zero-one law).

General remarks about the concept of independence.

1) Independence of functions of independent random variables. Given the independence of random variables one may deduce various propositions that are fairly obvious and also in full intuitive agreement with the idea of independence. For example, a function of and a function of , , are independent random variables. Functions of other types may be independent only if certain additional assumptions are made; such independence may serve in the definition of various classes of distributions. For example, if are independent, identically distributed and have a normal distribution, then the functions

(4)

and

(5)

(these are statistical estimators for the expectation and the variance of the , respectively) are independent random variables. The converse is also true: If are independent and identically distributed and if the functions (4) and (5) are independent, then the are normally distributed. In exactly the same way, if it is known that are independent and identically distributed, that the two linear forms

are independent random variables, that , and that none of the coefficients , vanishes, then all the are normally distributed. (This kind of theorem can be used to deduce, under minimal assumptions, say, Maxwell's law for the distribution of molecular velocities.) The above propositions are examples of what are known as characterization theorems, and were most thoroughly studied by Yu.V. Linnik and his school.

2) Existence of independent random variables on a given probability space. If the set of elementary events consists of three elements, each of which is assigned probability , then there do not exist non-constant independent random variables on . Letting the probability space be the interval with Lebesgue measure , then, given any sequence of distribution functions one can define measurable functions on that are independent random variables with respect to and such that

The simplest example of this kind of statistically-independent functions on is furnished by the signs of the binary decomposition of an , , or of the related Rademacher functions:

It should be noted that the existence of some probability space on which one can define independent random variables with given distributions is a corollary of Kolmogorov's theorem on probabilities in infinite-dimensional spaces (see [3], Chapt. III, Sect. 4).

3) Independent random variables as a source of other schemes. Let be a sequence of independent random variables and set

then one obtains a sequence of random variables forming a Markov chain. A similar procedure will yield a Markov process, for example, beginning with a Wiener process and using a stochastic differential equation. Starting with Gaussian random measures with independent values and using the Fourier transform, one can construct Gaussian stationary stochastic processes, etc.

4) Weak dependence. The asymptotic laws of probability theory that are established for sequences of independent random variables can usually be extended to sequences of so-called weakly-dependent variables, i.e. to sequences in which there is a suitably measured dependence between "distant" segments of the sequence that is "small" (in the simplest cases, these may be sequences of -dependent random variables, where and are independent if ; or sequences of random variables forming an ergodic Markov chain (cf. Markov chain, ergodic); etc.). One of the main methods for proving theorems of this type is reduction to the situation of independence.

5) Independence in number theory. Let and be two relatively-prime numbers. Let be a natural number, and suppose that a number between 1 and is chosen at random (the probability of each being chosen is assumed to be ). Let () be the event that the chosen number is divisible by (by ). Then

and if one lets , then the events and become "almost independent" . A much more profound proposition is the following: Letting , one can choose such that the events (where denotes divisibility by the -th prime) are jointly "almost independent" ; this proposition provides the basis for studying the value distribution of arithmetic functions (see Number theory, probabilistic methods in). There are also other branches of number theory in which the idea of independence plays an explicit or implicit part.

6) For testing of hypotheses of using independence results of observations, see Statistical hypotheses, verification of.

References

[1]	A.A. Markov, "Wahrscheinlichkeitsrechung" , Teubner (1912) (Translated from Russian)
[2]	A.N. Kolmogorov, "Foundations of the theory of probability" , Chelsea, reprint (1950) (Translated from Russian)
[3]	A.N. Kolmogorov, "The theory of probability" , Mathematics, its content, methods and meaning , 4 , Amer. Math. Soc. (1963) pp. Chapt. 6 (Translated from Russian)
[4]	M. Kac, "Statistical independence in probability, analysis and number theory" , Math. Assoc. Amer. (1963)
[5]	W. Feller, "An introduction to probability theory and its applications" , 1–2 , Wiley (1957–1971)

References

[a1]	A. Rényi, "Foundations of probability" , Holden-Day (1970)