Computer systems increasingly take part in decisions about people, and this section follows how those decisions can go wrong and what can be done about it. Its organizing idea is a three-part model of where bias enters a system: through its data, through its design, and through the way it is used. After establishing why the stakes are high, it takes each of those sources in turn, looks at two commercial examples in which several of them combine, and closes with the design practices that respond to them.

Why the stakes are high#

Automated decision systems score or rank people in hiring, lending, benefits, and criminal justice. These are high-stakes settings, because being denied a job, a loan, or release can have lasting consequences, so errors cause serious harm. That is why such systems should allow human review, explanation, and challenge: an applicant who can't understand a decision also can't contest it. Overall performance can be reassuring in misleading ways, because high overall accuracy can hide large differences in error rates between groups. A system that is right 95 percent of the time may be wrong far more often for one group of applicants, so the useful question is whether its errors are evenly shared.

When errors are not evenly shared, the term that applies is algorithmic bias, a systematic pattern of errors or outcomes that unfairly disadvantages particular groups. Systematic is the important word, since occasional mistakes are expected in any system and a recurring pattern calls for investigation. Bias can arise from data, from design choices, or from use in a setting the system wasn't designed for. Algorithms reflect human choices about data and goals, so "it's just math" isn't a defense.¹

Bias from data#

Data is the most common place to look first, because a model can learn only from what it's shown. Models perform worse for groups underrepresented in training data, as when speech recognition trained on a narrow range of speakers struggles with others, and they reproduce discrimination recorded in historical data, as when a model trained on past hiring repeats a past preference. Collecting more data from the same source preserves the imbalance, because the problem lies in who the data includes and not in how much of it there is. Representative data reflects the people a system will serve, so investigate the training data when performance differs across groups.²

An imbalance in the data can also grow over time in response to the system itself. In a feedback loop, a system's output shapes its future training data. More patrols produce more recorded arrests in an area, which appear to justify more patrols. Students denied placement in an advanced course have less chance to produce evidence of readiness for it. In this way, a system's decisions can make it look accurate, because it is partly generating the pattern that confirms it. Some exploration, such as showing less-viewed videos and measuring the response, adds data that its own choices wouldn't generate.

Bias from design#

The second source lies in what builders decide to ask of the data, starting with what the system should predict. A well-known example is predicting medical need from past spending: if some patients have received less care, they have spent less, and the system mistakes unequal care for lesser need. The choice of a target, made before any model was trained, built that unfairness in.

The choice of input features matters too, and a common but inadequate fix shows why. Removing a sensitive column doesn't solve the problem, because of proxy variables. A proxy variable is a feature that stands in for a protected characteristic, as a postal code can stand in for race where housing is segregated. Deleting the characteristic therefore doesn't necessarily remove bias, because the same information is still present in a neutral-looking field. This is also why comparing outcomes across groups, such as selection rates, reveals the problem better than checking the inputs.

Bias in use#

A system that was built well can still cause harm once it's in use, in two main ways. One is context: a system put to work in a setting it wasn't designed for may meet people and conditions that its builders never considered, and its results there can be unfair even if they were sound where it was developed. The other is how people respond to it. Automation bias is the tendency to accept a computer's output with too little scrutiny, even against other evidence. It's easy to fall into because computers do calculate correctly, but correct arithmetic doesn't ensure correct inputs or a correct model. A teacher handed a placement report should weigh it alongside what else they know about the student. Forming an independent judgment before seeing the output reduces this effect, since the output can't anchor a judgment that's already been recorded.

Two cases in which these combine#

Hiring shows several of these sources operating at once. Applicant tracking systems collect and screen job applications, and they can help or hurt. Masking personally identifiable information, such as names and addresses, can reduce reviewer bias at early stages. On the other hand, exact keyword filters can screen out qualified people who describe a skill differently, and training on current employees can encode existing preferences, since the model comes to favor backgrounds like those already there. When a feature turns out to be unrelated to ability and to affect some groups more than others, the appropriate response is to stop relying on it and review its effects on applicants.

Marketplace ranking adds a different source of tilt, the platform's own interests. Commercial platforms may rank results according to their own interests, including payments for placement or favoring their own brands, as well as relevance. Sellers can pay to appear higher, so a top position isn't proof of quality. Disclosure and independent reviews help users, which means a platform can reasonably be asked to say when commissions affect the order of results, and a shopper can filter out sponsored listings and compare reviews elsewhere.

Designing for inclusion#

Each of these failures suggests a response, and together those responses amount to what's called inclusive design. It involves diverse users throughout development, because barriers are hard to fix late and a development team may not experience them itself. It tests performance across groups, so that a good average doesn't hide a failure for older users or others. It explains decisions and offers human review and appeal, which answers the problem of people who can't understand or challenge an outcome. And it provides a way out of the automated path altogether: a human contact option helps people an automated system can't serve. One part of this work has been developed in particular detail, with established features, standards, and legal requirements: accessibility for people with disabilities, the subject of 0001.e.

• automated decision system: a system that scores or ranks people in settings such as hiring, lending, benefits, and criminal justice
• algorithmic bias: a systematic pattern of errors or outcomes that unfairly disadvantages particular groups
• representative data: data that reflects the people a system will serve
• proxy variable: a feature that stands in for a protected characteristic
• feedback loop: a pattern in which a system's output shapes its future training data
• automation bias: the tendency to accept a computer's output with too little scrutiny, even against other evidence
• applicant tracking system: software that collects and screens job applications
• inclusive design: involving diverse users, testing across groups, explaining decisions, and offering human review and appeal

Practice: practice questions for 0001.d

1. MWC2, Classification: Features: train/test splits, evaluation, feature engineering, and data cleaning. ↩

2. MWC2, Estimation: fitting models and overfitting. ↩