Section 0003.b described two kinds of memory inside a computer: RAM, fast but volatile, and storage, slower but lasting. That contrast is the first step of a larger hierarchy. Moving outward from the processor, data can sit in registers, in RAM, on a drive inside the device, on a device elsewhere on the local network, or on servers across the Internet, and each step outward generally trades speed and direct control for capacity, sharing, and independence from any one device. This section follows that outward path, starting with how storage is measured and the devices inside a computer, then comparing local and remote storage. Once data lives in more than one place, a second question arises: how the copies relate to one another. The rest of the section takes up the main answers, which are copies kept in sync, copies kept for redundancy, data split into parts, and copies placed near users.

Measuring storage#

Storage is measured in bytes, and a byte is 8 bits (0003.h explains bits). The larger units each step up by roughly a factor of a thousand: a kilobyte is about a thousand bytes, a megabyte about a thousand kilobytes, a gigabyte about a thousand megabytes, and a terabyte about a thousand gigabytes. Some sense of scale helps. Text is measured in kilobytes, a photo in megabytes, and a two-hour high-definition video in gigabytes. Because each unit is a thousand times the one before, estimates are mostly a matter of counting thousands. Two thousand recordings of 500 megabytes each come to 1,000,000 megabytes, which is about 1,000 gigabytes, or 1 terabyte. A 64-gigabyte card holds about 64,000 megabytes, so it fits about 16,000 photos of 4 megabytes each. When comparing sizes written in different units, convert first: 64,000 kilobytes is only about 64 megabytes, smaller than a 900-megabyte file.

Network speeds add one trap. They are usually quoted in bits per second, while file sizes are given in bytes. Since a byte is 8 bits, a 100-megabyte file is 800 megabits, so over a 100-megabit-per-second connection it takes about eight seconds, not one.

Storage devices#

Inside a computer or attached to it, three kinds of device are common. A hard disk drive (HDD) stores data magnetically on spinning platters, read and written by a head that moves to each requested location. Hard drives offer a low cost per unit of storage, which suits a large, rarely used archive. Their moving parts are also their weakness. Moving the head takes time, which slows random access, where data are read from scattered locations, and a fall can damage the platters or head. A solid-state drive (SSD) stores data in flash memory, which keeps data without power and has no moving parts. With no head to move, scattered reads are much faster, and with nothing mechanical to damage, an SSD is more durable. A sensible arrangement often uses both: hard drives for the archive and solid-state drives for files in constant use. Removable media such as USB drives are portable and need no network, which makes them useful for carrying files to a place where the network is blocked, but a small device is easy to lose.

Local and remote storage#

Where data is kept is a choice with trade-offs on both sides. Local storage, on a device the owner holds or on a server they run, offers fast access without an Internet connection and direct control over the data. A school whose files are on its own server can reach them during an Internet outage, since no outside connection is needed. The other side of control is responsibility: the owner handles maintenance and backups, and data can be lost with the device. A photographer whose only copy of a collection is on one laptop risks losing it permanently if the laptop fails or is stolen, and the remedy is making additional copies. Holding the device gives control, but the owner also takes on theft, loss, and upkeep, so the security of local data depends on those practices, not on location alone. Local storage suits an organization with staff to maintain it, such as a clinic that keeps an on-site storage server its own staff maintain and back up.

Cloud storage keeps data on remote servers reached through a network. Each of a user's devices connects to the copy held on the provider's servers and syncs with it, so a change made on a laptop reaches the phone, and files can be reached from any location and shared with others. Because the remote copy is separate from the user's devices, it survives losing a device: a student whose laptop is stolen can still reach coursework that was also held in the cloud. The trade-offs come from depending on others. Cloud storage depends on connectivity, so an outage blocks access even though the data are intact. It depends on a provider's security, terms, and continued service, so sensitive data such as medical or student records calls for a review of both security and terms before convenience is considered.¹ And its costs continue: recurring subscription fees, and lock-in, the difficulty of moving data to another provider, can outweigh a first year's savings.

Between the device and the cloud sits a middle option. Network-attached storage (NAS) is a dedicated device that provides shared file storage over a local network. Because local network speeds are often much faster than Internet connections, a NAS moves huge files, such as video projects, faster than cloud storage, since they never cross the Internet connection. Its performance depends on traffic on the local network and on demand on the device. Like other local storage, it is a single location, so a fire or burglary can destroy everything on it unless a copy is kept elsewhere.

Keeping copies consistent#

Once data is in more than one place, the copies have to be kept consistent. Synchronization carries a change made on one copy to the others, so that all copies stay consistent across devices. Trouble comes when copies are out of contact. If two people edit the same document while offline, their copies diverge, and when they reconnect the conflicting edits must be merged or resolved. A simple rule called "last write wins" keeps only the copy saved most recently, which means the changes in the other version disappear with no warning to their author. Timing is a trade-off as well: syncing less often uses less bandwidth during the day, but copies diverge for longer, so people work from outdated copies and conflicts become more likely. One more property of sync is easy to miss. A synchronized copy reproduces deletions as well as additions, so it isn't independent, and sync isn't a backup. A backup is an independent copy that a mistake on the original doesn't reach.

Redundant copies#

Synchronization keeps copies consistent for convenience, and redundant storage keeps copies for availability. Redundant arrays of independent disks (RAID) keep data across several drives, so that data remain available if one drive fails. In mirroring, one common form, a second drive keeps a complete copy of the first, so a server can continue with no loss if one drive stops working. Replication does the same across locations, so that a flood shutting down one data center for a week leaves the data available at another. These are the storage version of the redundancy described in 0003.d, and they have costs of their own. Distant replication can slow writes, because each write waits for its copy to cross a network and be confirmed at the second site. More importantly, redundancy copies every change at once, including accidental deletions and file corruption, which reach both drives at the same moment. Like sync, RAID and replication therefore don't replace backups, and they don't address other threats such as stolen passwords.

Splitting data across devices#

Some datasets are too large or too busy for one device, and the response is to divide them instead of copying them. Partitioning, or sharding, divides a database across devices, with each piece of data living in one place. A good partitioning generally groups data likely to be used together, such as storing each region's customers together on their own server, or each year's records separately, so that most queries read far less data, since older years needn't be searched. Two problems limit this. Uneven keys overload some partitions: dividing usernames by first letter sounds even, but letters aren't equally common among usernames, so the servers for common letters receive most of the data and traffic. And queries that cross partitions are slower, because a report that needs data from every partition must gather and combine results from many separate devices.

Copies near users#

The last arrangement places copies close to the people who need them. A content delivery network (CDN) stores copies of popular content on servers in many regions and answers each request from a nearby location, which reduces latency, the delay in data travel. A company whose videos load slowly for European viewers can fix this by storing copies on servers in or near Europe. Because most requests are served by those copies, a CDN also reduces the load on the origin server, the one that holds the original. Its copies are caches, which persist until they expire or are replaced, so after a file is updated some regional servers may briefly keep delivering the earlier copy. CDNs therefore suit popular static content, such as a product video watched by customers around the world, and not private or rapidly changing data. Latency, load, and capacity are themselves among the factors that shape how networks perform, which the next section examines.

• byte: 8 bits; kilobyte, megabyte, gigabyte, and terabyte are each roughly a thousand times the one before
• hard disk drive (HDD): storage that records data magnetically on spinning platters, with low cost per unit of storage
• solid-state drive (SSD): storage that uses flash memory, with no moving parts, so it is faster and more durable
• removable media: portable storage such as USB drives
• local storage: data kept on a device or server the owner controls
• cloud storage: data kept on remote servers reached through a network
• lock-in: the difficulty of moving data to another provider
• network-attached storage (NAS): a dedicated device providing shared file storage over a local network
• synchronization: keeping copies of data consistent across devices
• backup: an independent copy used to restore data after loss
• RAID: redundant arrays of independent disks, which keep copies of data across drives
• mirroring: keeping a complete copy of one drive on another
• replication: keeping copies of data in separate locations
• partitioning (sharding): dividing a database across devices, grouping data likely to be used together
• content delivery network (CDN): servers in many regions that store copies of popular content near users
• origin server: the server that holds the original content

Practice: practice questions for 0003.f

1. MWC2, Banjo app: a teacher portal case study, including requirements, components, and FERPA. ↩