Information theory

Information theory is the study of how we can measure, store, and share information. It started with work by Claude Shannon in the 1940s, with help from Harry Nyquist and Ralph Hartley. Though it began with phone and radio systems, it now connects mathematics, statistics, and computer science. It helps many fields like electrical engineering and neurobiology.

One easy example is flipping a fair coin. Before you look, you don’t know if it will land on heads or tails, so you are missing some information. After it lands, you learn that information. For a fair coin, this amount of information is called 1 bit.

Information theory has helped create many modern technologies. It helps us make files smaller for ZIP files, fix mistakes in DSL connections, and made the Voyager missions, compact discs, mobile phones, the Internet, and artificial intelligence possible. It also plays roles in areas like cryptography, bioinformatics, and even the study of black holes.

Overview

Information theory, created by Claude Shannon, studies how we handle and use information when we are not sure about something. Think of trying to send a message over a radio with lots of static — Shannon showed we can send messages with very few errors if we understand the radio's limits.

This field also helps us make data smaller, like zipping a file, and fix mistakes when messages get mixed up. These ideas are used in secret codes and puzzles to protect information.

Historical background

Main article: History of information theory

Information theory started with a big paper by Claude Shannon in 1948. The paper was called "A Mathematical Theory of Communication." It showed new ways to understand how we send, store, and measure information.

Before Shannon, other people like Harry Nyquist and Ralph Hartley had looked at some ideas about information. But Shannon brought all these ideas together into one strong theory.

Shannon showed how to measure information using a unit called the bit. He also explained how to make communication better and fix problems when information travels through noisy or imperfect channels. His work has helped many areas, from telecommunications to computer science.

Quantities of information

Main article: Quantities of information

Information theory studies how we measure and share information. It uses math to look at how much information we can send. One key idea is called entropy. Entropy helps us understand how hard it is to guess what will happen next.

We often describe information in units called bits. For example, if we send a bit that can be either a 0 or a 1, entropy tells us how surprising that number might be. When all results are equally likely, entropy is highest, meaning it’s hardest to predict.

Another idea is mutual information. This looks at how two things are related. It helps us see how well we can send information even when there is noise or interference. The way we measure this depends on the kind of numbers we use, like bits.

Coding theory

Main article: Coding theory

Coding theory is an important part of information theory. It helps us learn how to store and send information correctly. Information theory tells us how many bits we need to describe data. This is called information entropy.

There are two main parts to coding theory:

Data compression (source coding): This makes data smaller. There are two types:
- Lossless data compression: The data is made smaller but can be restored perfectly.
- Lossy data compression: Some small details are lost, but the data is still useful. This is called rate–distortion theory.
Error-correcting codes (channel coding): This adds extra information to data so that mistakes can be fixed when the data is sent over a connection with noise.

Coding theory helps split data into parts for compression and sending. This works best when one person sends to one person. With more senders or receivers, it becomes more difficult.

Source theory

Any process that creates messages can be a source of information. A memoryless source is one where each message is random and not connected to the others. These sources are studied in information theory.

Channel capacity

Main article: Channel capacity

When sending messages, we often face noise or problems that change the message. A channel is like a path for sending messages, which can sometimes make mistakes.

The goal is to send as much information as possible over this channel without many errors. The maximum amount of information that can be sent safely is called the channel capacity. This depends on how noisy the channel is.

Different types of channels have different capacities. For example:

A channel with Gaussian noise has a certain capacity, described by the Shannon–Hartley theorem.
A binary symmetric channel (BSC) can flip bits with some chance.
A binary erasure channel (BEC) can sometimes erase bits completely.

Fungible information

Fungible information is information where how it is encoded doesn’t matter. This is the kind of information most scientists study. It is sometimes called speakable information.

Applications to other fields

Information theory is used in many different areas. It helps us understand how living things share and process information, like how the brain and body work together. It also helps keep information safe, like in secret codes.

Other uses include making better random numbers for computers, improving how we find oil underground, and studying how we think and make decisions. Information theory helps scientists search for signs of life beyond Earth and understand complex natural phenomena like black holes.