The previous section described the hardware of a network and how it can be arranged. For devices made by different companies to exchange data across that hardware, they have to follow shared rules, and a protocol is a set of such rules, saying how messages are formatted, addressed, sent, and acknowledged. The Internet uses many protocols, and they are easiest to understand as a stack of layers, each doing one job and relying on the layer below it. The layers also correspond to levels of granularity. At the bottom, signals carrying bits move across a single link, such as a Wi-Fi connection or a cable. One level up, packets travel from network to network across the Internet. Above that, a connection between two programs carries a whole stream of data reliably, or deliberately doesn't, and at the top, applications such as the web and email exchange requests, pages, and messages. This section starts with the idea that makes the Internet work, packet switching, then lays out the layers, climbs through them from the link to the applications, and ends by using the layers to diagnose problems.

Packet switching#

The telephone network of the twentieth century used circuit switching: a call reserved a dedicated path between two phones for its whole length, holding that capacity even during silences. The Internet instead uses packet switching.¹ Data are broken into small pieces called packets, and each packet is sent separately. Every packet has a header, which says where it is going and carries information for putting the pieces back in order, and a payload, the data being delivered. Each router along the way makes its own forwarding decision from the header, so packets from one message may take different routes and arrive out of order, and the receiving end reassembles them using the header information.

This design has two large advantages. First, the network can route around failures: if a router fails partway through a transfer, later packets can be sent along a different route. Second, many users can share the same links. When a neighborhood's households share one cable, each household's data is split into packets that take turns on it, whereas a dedicated circuit would hold capacity for one conversation even when it is idle. Sharing works economically because most users send data in bursts, loading a page and then reading it, so their peaks rarely coincide and a shared link spends little capacity on idle users. The price is that performance can vary with load, a point 0003.g develops.

The layered model#

Packets are only part of the story. Something has to carry them across each physical link, something has to route them between networks, something has to make sure they all arrive, and something has to give them meaning to an application. The Internet's protocols divide these jobs into four layers, each relying on the one below it:

Layer	Job	Examples
Application	What programs say to each other	HTTP, DNS, SMTP, IMAP, FTP
Transport	Delivering data between programs, reliably or quickly	TCP, UDP
Internet	Addressing and routing packets between networks	IP, ICMP
Link	Moving data across one local link	Wi-Fi, Ethernet

When a browser requests a page, the request passes down the stack, from HTTP to TCP to IP to Ethernet. At each layer, a header is added that wraps the data handed down from the layer above, so the HTTP request becomes the payload of a TCP segment, which becomes the payload of an IP packet, which is carried inside an Ethernet frame. The receiver unwraps the headers in reverse order. The benefit of layering is that each layer can ignore how the others work. A browser uses HTTP without handling routing, which is IP's job, while TCP manages reliable delivery. Lower layers can also change without affecting the applications above: a school that replaces its cabled network with Wi-Fi changes the link layer, and every web application keeps working unchanged. The sections below climb the stack from the link layer up.

The link layer: Wi-Fi and hardware addresses#

The link layer moves data across a single local connection. Ethernet does this over cables, and Wi-Fi connects devices wirelessly using radio signals. Because Wi-Fi uses radio, walls and distance weaken its signal, which is a separate matter from the speed of the Internet connection behind it. Wi-Fi also divides its radio frequencies into channels, and networks sharing a channel compete for airtime, so in a crowded building with many networks, moving to a channel with less interference helps.

A wireless network announces a name, its service set identifier (SSID). The SSID only labels the network. It is separate from the Wi-Fi password, which lets devices join, and from the router's administrator password, which controls changes to the router's settings. A router whose default administrator password was never changed can be reconfigured by anyone who knows that default, whatever the Wi-Fi password is.

Within a local network, data are delivered using MAC addresses, hardware identifiers assigned to each network interface (the hardware that connects a device to a network). The switches described in 0003.d forward data using these addresses. A MAC address identifies an interface, while an IP address, described next, depends on the network a device is currently connected to. Some networks admit only approved MAC addresses, a measure called MAC filtering, but this is weak by itself, because MAC addresses can be changed in software, so a device with a copied address looks like the approved one.

The Internet layer: IP and its addresses#

The Internet layer moves packets between networks. Internet Protocol (IP) addresses and routes packets, and routers forward each packet toward its destination network based on the IP address in its header. IPv4 uses 32-bit addresses written as four numbers separated by dots, such as 192.168.1.20. Each number is 8 bits, so each must be between 0 and 255. Thirty-two bits allow about 4.3 billion addresses, which proved too few for a world of phones, laptops, and connected devices, and IPv6, with 128-bit addresses that vastly expand the space, was developed largely in response to IPv4 address exhaustion.

One way IPv4 has stretched its supply is with private addresses. Addresses such as 192.168.x.x are private: they are reused on many local networks, and the router translates them to a public address for Internet traffic, so the duplicates never conflict. This shows something more general about IP addresses. Networks give addresses to the devices that join them, and routing depends on where a device is connected, so an IP address reflects the current network and changes as a device moves, from home to school, for instance. Many users or devices may also share one address. An IP address therefore identifies a location on the network, not a person.

Assigning these addresses by hand doesn't work well: giving two devices the same address causes a conflict. Dynamic Host Configuration Protocol (DHCP) solves this by having a server automatically assign available IP addresses to devices as they join a network. Because DHCP acts when devices join, a DHCP failure affects new arrivals first, while laptops that already hold addresses keep working. A device that needs a stable address, such as a shared printer, can have a fixed address reserved for it on the DHCP server, which keeps automatic assignment for everything else. And as 0003.c noted, a device showing a self-assigned 169.254 address got no address from DHCP at all.

The Internet layer also carries messages about the network itself. Internet Control Message Protocol (ICMP) carries diagnostic and error messages, and two familiar tools use it to test connections. ping sends a message to a host and reports whether it responds and the round-trip delay, the time for the message to get there and back. That delay is a different property from capacity, so a ping time says nothing about how much data a connection can carry. Many servers and firewalls are configured to ignore these diagnostic messages, so no reply doesn't prove that a host is down. traceroute lists each hop, each router along the path to a destination, with the delay to each. Stars in its output mean no reply, so output that stops after a hop suggests that packets are failing or being blocked somewhere beyond it. If the delay to your own router is low but delay jumps farther along, the local network is fine and the slowness lies along the path to the site.

The transport layer: TCP and UDP#

IP delivers packets on a best-effort basis: some may be lost, and they may arrive out of order. The transport layer decides what to do about that, and the Internet offers two protocols that make opposite choices. Both use port numbers to separate the different applications running on one host, so that web traffic and mail traffic arriving at the same address reach the right programs.

Transmission Control Protocol (TCP) provides reliable, ordered delivery. Before any application data flows, the two ends establish a connection with a three-way handshake, an exchange of three messages in which they agree to communicate. TCP then numbers the data it sends, the receiver sends acknowledgments confirming receipt, and if an acknowledgment doesn't arrive within a timeout, the sender retransmits. The numbering lets the receiver restore the original order. Web pages and email need this, since a page or message with missing or scrambled pieces is useless. TCP also slows down when it detects signs of congestion, such as lost packets, and then speeds up again, which helps keep shared networks stable. TCP's guarantees don't include secrecy, which is provided separately, and they cost time, so TCP isn't the faster of the two.

User Datagram Protocol (UDP) sends packets faster, with lower overhead and no delivery confirmation. It does nothing about loss or order, so losses go unnoticed unless the application checks for them, though UDP packets still have a header with addresses and ports. This suits real-time applications, where low delay matters more than completeness. In a live video call, audio that is resent would arrive too late to be useful, so waiting for retransmission would only add delay. In an online game, a newer position update soon replaces a lost one, so resending has little value. UDP also suits DNS lookups, described below, which are small queries and replies. Applications that use UDP but need some reliability can add their own sequence numbers and acknowledgments.

The application layer: names, the web, and email#

At the top of the stack, application protocols define what programs actually say to each other. The first thing most connections need is a name lookup. People use names like www.buffalo.edu, but IP routes by address, so the Domain Name System (DNS) translates domain names into IP addresses. Registering a domain name also creates a public record of who registered it, and a WHOIS lookup retrieves that record, which can help in judging who is behind a site, though many registrants now pay to keep their details private. DNS usually uses UDP on port 53. Its hierarchy begins at the end of a name: in mail.dept.buffalo.edu, the top-level domain edu comes first, then buffalo within it, and so on leftward. Each new name needs a lookup before a connection can start, so a slow lookup adds noticeable delay before a page begins to load. Answers are cached, stored for a set time and reused, which speeds later lookups but means that changes to a record spread gradually, as old answers remain in caches until they expire. DNS is also a point of attack. If its records are maliciously altered, users on an affected network can be redirected to the wrong IP address for a website even when they type its name correctly. A site that loads by its IP address but not by its name points to DNS as the problem.

Hypertext Transfer Protocol (HTTP) transfers web content through requests and responses: a client sends a request, and a server sends back a response.¹ Plain HTTP is unencrypted, so anyone capturing the traffic can read it. HTTPS adds encryption and server authentication using Transport Layer Security (TLS), which encrypts traffic in transit and uses certificates to tie the encryption to a domain. Its cost is a slightly slower connection setup, because the TLS handshake adds extra steps to verify the server and establish encryption, and that cost is generally outweighed by the benefits. A browser's padlock indicates an encrypted connection, not a trustworthy operator, since phishing and malicious sites can use HTTPS too. For a site with a login form, the fix for data readable on the way is enabling TLS on the server and redirecting all requests to HTTPS. Hashing stored passwords or requiring stronger ones does nothing for data in transit. The whole session needs protection, too: a site that returns to HTTP after login exposes its session cookies, the tokens that keep a user logged in, which can be captured and used to take over accounts.

An HTTP request names a method, saying what the client wants done. GET requests data, and POST submits it.¹ When a student submits a guestbook entry and then sees it on the page, the POST submitted the entry and the later GET only displayed it. The difference has consequences. Following a link sends a GET, which automated tools such as search-engine crawlers assume is safe to repeat, so a link that deletes data can be triggered by a crawler, and deletions need another method such as POST or DELETE. Sending a password in a GET request puts it in the URL, and although HTTPS protects the URL in transit, URLs are often kept in browser history and server logs, so passwords belong in a POST body over HTTPS. Other methods exist as well: PUT, DELETE, OPTIONS, and TRACE. OPTIONS asks which methods a server allows for a resource, without changing anything. Repeating a PUT has the same effect as sending it once, unlike POST, which may create a new entry each time. These methods exist because HTTP was designed more broadly than the browsing that now dominates its use, for a collaborative network, a shared space where people would edit documents together as well as read them. Most public services now limit these methods.

Each response carries a three-digit status code, grouped by its first digit.¹ Codes in the 200s mean success, the 300s mean redirection, the 400s mean client errors, and the 500s mean server errors. A few codes are worth knowing individually. 301 signals a lasting move and sends users and search engines to the new address, which makes it the right response when a site reorganizes its pages. 401 means the user needs to log in. 404 Not Found often means a mistyped path. 503 means the service is temporarily overloaded or unavailable, so the users did nothing wrong and a later attempt should work. Because an error response can still have a body, a script fetching data should check the status code before using a response.

Email and file transfer use their own protocols. Simple Mail Transfer Protocol (SMTP) sends email and relays it between mail servers, and IMAP retrieves it from a server, which is why a mail program asks for separate sending and receiving settings. IMAP keeps mail on the server and keeps its state in sync across devices, so a message read on a phone shows as read on a laptop. Each server that handles a message typically adds a Received: line to its header, so several of them show that the message passed through multiple mail servers. An open SMTP relay, one that forwards mail from anyone, gets abused for spam. File Transfer Protocol (FTP) transfers files, but plain FTP sends credentials and files unencrypted, so anyone capturing the traffic can read them.

Troubleshooting with the layers#

The layered model also gives troubleshooting a systematic order. Because each layer relies on the ones below, a problem can be located by working upward from the lowest layer: confirm the Wi-Fi signal or cable connection, then whether the device has an IP address, then whether it can reach its router and beyond with ping or traceroute, then whether names resolve through DNS, and finally whether the application itself responds. Each check that passes clears a layer and narrows the search above it. Protocols determine how data moves, but data also has to be kept somewhere between transfers, and the next section turns to where data is stored.

• protocol: a set of rules for how devices format, address, send, and acknowledge messages
• packet switching: breaking data into packets that are routed separately and reassembled at the destination
• packet: a unit of data with a header and a payload
• header: the part of a packet that says where it is going and how to reassemble it
• link layer: moves data across one local link (Wi-Fi, Ethernet)
• Internet layer: addresses and routes packets between networks (IP)
• transport layer: delivers data between programs (TCP, UDP)
• application layer: defines what programs say to each other (HTTP, DNS, SMTP)
• Wi-Fi: wireless networking using radio signals
• service set identifier (SSID): a wireless network's name
• MAC address: a hardware identifier used for delivery within a local network
• Internet Protocol (IP): the protocol that addresses and routes packets between networks; IPv4 uses 32-bit addresses and IPv6 uses 128-bit addresses
• private address: an address such as 192.168.x.x, reused on local networks and translated at the router
• Dynamic Host Configuration Protocol (DHCP): automatically assigns IP addresses to devices joining a network
• Internet Control Message Protocol (ICMP): carries diagnostic and error messages, used by ping and traceroute
• port number: a number that separates applications on one host
• Transmission Control Protocol (TCP): reliable, ordered delivery using a handshake, acknowledgments, and retransmission
• User Datagram Protocol (UDP): fast, low-overhead delivery with no confirmation
• Domain Name System (DNS): translates domain names into IP addresses
• WHOIS: a lookup that retrieves the public registration record for a domain name
• Hypertext Transfer Protocol (HTTP): transfers web content through requests and responses
• HTTPS: HTTP with encryption and server authentication using TLS
• HTTP methods: GET requests data, POST submits it, and PUT, DELETE, OPTIONS, and TRACE serve other purposes
• status codes: 200s success, 300s redirection, 400s client errors, 500s server errors
• SMTP: sends and relays email
• IMAP: retrieves email, keeping it on the server and in sync
• File Transfer Protocol (FTP): transfers files

Practice: practice questions for 0003.e

1. MWC2, Riddles: HTTP requests, responses, verbs, and status codes. ↩ ↩2 ↩3 ↩4