Information is a very abstract concept. People often say that there is a large amount of information or a small amount of information, but it is difficult to say clearly how much information there is. For example, how much information does a 500,000-word Chinese document have?

It was not until 1948 that Shannon put forward the concept of "information entropy" and solved the problem of quantitative measurement of information. The word information entropy is borrowed from thermodynamics by C.E.Shannon, and the thermal entropy in thermodynamics is a physical quantity indicating the disorder degree of molecular state. The concept of agricultural information entropy is used to describe the uncertainty of information sources.

Claude elwood shannon, the father of information theory, first expounded the relationship between probability and information redundancy in mathematical language.

Chinese name

Information entropy

Foreign name

Information entropy

presenter

C.e. Shannon

time

1948

use for reference

The concept of thermodynamics

quick

navigate by water/air

Basic content information meaning

Put forward theory

Shannon, the father of information theory, pointed out in the paper Mathematical Theory of Communication published in 1948 that any information is redundant, and redundancy is related to the probability or uncertainty of each symbol (number, letter or word) in the information.

Shannon used the concept of thermodynamics for reference, called the average information excluding redundancy "information entropy", and gave a mathematical expression to calculate the information entropy.

basic content

Usually, what symbol a source sends out is uncertain and can be measured according to its probability of occurrence. High probability, many opportunities and little uncertainty; On the contrary, the uncertainty is great.

The uncertainty function f is a decreasing function of probability p; The uncertainty produced by two independent symbols should be equal to the sum of their respective uncertainties, that is, f (P 1, P2) = f (P 1)+f (P2), which is called additivity. The function f that satisfies these two conditions at the same time is a logarithmic function, that is

In the source, what we should consider is not the uncertainty of a single symbol, but the average uncertainty of all possible situations of this source. If the source symbol has n values: U 1…Ui…Un, the corresponding probability is: P 1…Pi…Pn, and the appearance of various symbols is independent of each other. At this time, the average uncertainty of the information source should be the statistical average (e)-logPi of the uncertainty of a single symbol, which can be called information entropy, i.e.