Acknowledgment: Richard Hamming's book, Information Theory andCoding, Prentice-Hall, New York (1985) and C. T. Mullis's unpublished notes have influenced our treatment of binary codes. The numerical experiment was developed by Mullis.

We use this chapter to introduce students to the communication paradigm and to show how arbitrary symbols may be represented by binary codes. These symbols and their corresponding binary codes may be computer instructions, integer data, approximations to real data, and so on.

We develop some ad hoc tree codes for representing information and then develop Huffman codes for optimizing the use of bits. Hamming codes add check bits to a binary word so that errors may be detected and corrected. The numerical experiment has the students design a Huffman code for coding Lincoln's Gettysburg Address.

Introduction

It would be stretching our imagination to suggest that Sir Francis had digital audio on his minde (sic) when he wrote the prophetic words

Sir Francis Bacon, 1623

...a man may express and signifie the intentions of his minde, at any distance... by... objects... capable of a twofold difference onely.

Nonetheless, this basic idea forms the basis of everything we do in digital computing, digital communications, and digital audio/video. In 1832, Samuel F. B. Morse used the very same idea to propose that telegram words be coded into binary addresses or binary codes that could be transmitted over telegraph lines and decoded at the receiving end to unravel the telegram. Morse abandoned his scheme, illustrated in Figure 1, as too complicated and, in 1838, proposed his fabled Morse code for coding letters (instead of words) into objects (dots, dashes, spaces) capable of a threefold difference onely (sic).

Figure 7.1. 

Generalized Coder-Decoder

The basic idea of Figure 1 is used today in cryptographic systems, where the “address a_i" is an encyphered version of a message w_i ; in vector quantizers, where the “address a_i" is the address of a close approximation to data w_i ; in coded satellite transmissions, where the “address a_i" is a data word w_i plus parity check bits for detecting and correcting errors; in digital audio systems, where the “address a_i" is a stretch of digitized and coded music; and in computer memories, where a_i is an address (a coded version of a word of memory) and w_i is a word in memory.

In this chapter we study three fundamental questions in the construction of binary addresses or binary codes. First, what are plausible schemes for mapping symbols (such as words, letters, computer instructions, voltages, pressures, etc.) into binary codes? Second, what are plausible schemes for coding likely symbols with short binary words and unlikely symbols with long words in order to minimize the number of binary digits (bits) required to represent a message? Third, what are plausible schemes for “coding” binary words into longer binary words that contain “redundant bits” that may be used to detect and correct errors? These are not new questions. They have occupied the minds of many great thinkers. Sir Francis recognized that arbitrary messages had binary representations. Alan Turing, Alonzo Church, and Kurt Goedel studied binary codes for computations in their study of computable numbers and algorithms. Claude Shannon, R. C. Bose, Irving Reed, Richard Hamming, and many others have studied error control codes. Shan- non, David Huffman, and many others have studied the problem of efficiently coding information.

In this chapter we outline the main ideas in binary coding and illustrate the role that binary coding plays in digital communications. In your subsequent courses in electrical and computer engineering you will study integrated circuits for building coders and decoders and mathematical models for designing good codes.

A paradigm is a pattern of ideas that form the foundation for a body of knowledge. A paradigm for (tele-) communication theory is a pattern of basic building blocks that may be applied to the dual problems of (i) reliably transmitting information from source to receiver at high speed or (ii) reliably storing information from source to memory at high density. High-speed communication permits us to accommodate many low-rate sources (such as audio) or one high-rate source (such as video). High-density storage permits us to store a large amount of information in a small space. For example, a typical 1.2 Mbyte floppy disc stores 9.6×10⁶ bits of information, whereas a typical CD stores about 2×10⁹ bits, enough for one hour's worth of high-quality sound.

Figure 7.2. 

Basic Building Blocks in a (Tele-) Communication System

Figure 1 illustrates the basic building blocks that apply to any problem in the theory of (tele-) communication. The source is an arbitrary source of information. It can be the time-varying voltage at the output of a vibration sensor (such as an integrating accelerometer for measuring motion or a microphone for measuring sound pressure); it can be the charges stored in the CCD array of a solid-state camera; it can be the addresses generated from a sequence of keystrokes at a computer terminal; it can be a sequence of instructions in a computer program. The source coder is a device for turning primitive source outputs into more efficient representations. For example, in a recording studio, the source coder would convert analog voltages into digital approximations using an A/D converter; a fancy source coder would use a fancy A/D converter that finely quantized likely analog values and crudely quantized unlikely values. If the source is a source of discrete symbols like letters and numbers, then a fancy source code would assign short binary sequences to likely symbols (such as e ) and long binary sequences to unlikely symbols (such as z ). The channel coder adds “redundant bits” to the binary output of the source coder so that errors of transmission or storage may be detected and corrected. In the simplest example, a binary string of the form 01001001 would have an extra bit of 1 added to give even parity (an even number of l's) to the string; the string 10110111 would have an extra bit of 0 added to preserve the even parity. If one bit error is introduced in the channel, then the parity is odd and the receiver knows that an error has occurred. The modulator takes outputs of the channel coder, a stream of binary digits, and constructs an analog waveform that represents a block of bits. For example, in a 9600 baud Modem, five bits are used to determine one of 2⁵=32 phases that are used to modulate the signal A cos(ωt+φ). Each possible string of five bits has its own personalized phase, φ , and this phase can be determined at the receiver. The signal Acos(ωt+φ) is an analog signal that may be transmitted over a channel (such as a telephone line, a microwave link, or a fiber-optic cable). The channel has a finite bandwidth, meaning that it distorts signals, and it is subject to noise or interference from other electromagnetic radiation. Therefore transmitted information arrives at the demodulator in imperfect form. The demodulator uses filters matched to the modulated signals to demodulate the phase and look up the corresponding bit stream. The channel decoder converts the coded bit stream into the information bit stream, and the source decoder looks up the corresponding symbol. This sequence of steps is illustrated symbolically in Figure 7.3.

Figure 7.3. 

Symbolic Representation of Communication

In your subsequent courses on communication theory you will study each block of Figure 1 in detail. You will find that every source of information has a characteristic complexity, called entropy, that determines the minimum rate at which bits must be generated in order to represent the source. You will also find that every communication channel has a characteristic tolerance for bits, called channel capacity. This capacity depends on signal-to-noise ratio and bandwidth. When the channel capacity exceeds the source entropy, then you can transmit information reliably; if it does not, then you cannot.

Perhaps the most fundamental idea in communication theory is that arbitrary symbols may be represented by strings of binary digits. These strings are called binary words, binary addresses, or binary codes. In the simplest of cases, a finite alphabet consisting of the letters or symbols s₀,s₁,...,s_M–1 is represented by binary codes. The obvious way to implement the representation is to let the i^th binary code be the binary representation for the subscript i :

(7.1)

The number of bits required for the binary code is N where

We say, roughly, that N=log₂M.

Octal Codes. When the number of symbols is large and the corresponding binary codes contain many bits, then we typically group the bits into groups of three and replace the binary code by its corresponding octal code. For example, a seven-bit binary code maps into a three-digit octal code as follows:

(7.3)

The octal ASCII codes for representing letters, numbers, and special characters are tabulated in Table 1.