Whether data will feed a model or answer a simpler question, its value is largely settled by the time it has been collected, because analysis can't repair what was never gathered properly. Data collection is best understood as a set of decisions made before any data exists, followed by responsibilities that continue after it does. This section begins with the plan that records those decisions. It then asks who the data represents, looks at three tools for gathering it (sensors, surveys, and web scrapers), and ends with what collectors owe the people in the data and how to judge what they have.

The plan: data collection protocols#

A data collection protocol states what is being collected, from whom, how it will be stored, who it will be shared with, and why. It is decided in advance, so that collection is consistent, ethical, and repeatable.¹ Each of those qualities answers a different problem. Consistency matters because two teachers measuring stream temperature can't combine their readings if one records Celsius at noon and the other Fahrenheit whenever convenient, so agreeing on units and times is what makes the data comparable. Ethics matters because a team that never settled where responses would be kept may find them on personal laptops and in the hands of a club. Repeatability matters because another researcher can repeat a study only if the questions, recruitment steps, storage plan, and intended uses are written down. A protocol's questions are different from the questions of later analysis, such as an average or a correlation, since a protocol asks things like who will have access. Stating a purpose also creates an obligation, so new uses should be disclosed. A district that collects students' web activity for safety and later uses it to evaluate teachers has gone beyond what it told families.

Who the data represents#

One decision in the protocol, from whom to collect, deserves special attention, because a mistake here undermines every conclusion and can't be seen in the data itself. Sampling bias occurs when the people or items sampled don't represent the population of interest. Polling the jazz band tells you about the jazz band and can mislead you about a whole school's views on a later start time. Online surveys miss people who rarely go online, even when those are the people a question concerns most, as when older residents are a library's most frequent patrons but rarely visit the town's website. Voluntary responses overrepresent strong opinions, which is why reviews pile up at one star and five stars. People who choose to use a fitness app may be more active than the population as a whole, so their step counts say little about the country. Collecting more of the same responses, or leaving a survey open longer, doesn't address the problem, because it doesn't change who is responding. What helps is reaching underrepresented groups through other channels, such as paper forms in the library, and using a random procedure to choose whom to ask where that is possible.

Three tools for gathering data#

Knowing whom or what to sample leaves the question of how, and the tools differ in what can go wrong with them. Sensors convert physical conditions to digital data at a sampling rate, which is how often a measurement is recorded. The rate must be frequent enough to capture the change of interest without creating needless data, so it depends on how quickly the thing you care about changes. Sampling too slowly misses brief events: an hourly reading of a cooling cup of coffee misses most of its rapid early change, a once-per-second sensor can't follow a ball that bounces twice per second, and detecting an irregular heartbeat requires far more frequent samples than reporting a weekly average. Sampling too fast has the opposite cost, as when a soil moisture sensor records every second and fills its storage with readings of something that changes over hours. Precision and units are separate from rate. A sensor that displays more digits still misses what happens between its readings.

Surveys collect from people, and the weak point moves from timing to wording. Questions should be clear and unbiased. A leading question pushes toward an answer ("Do you agree that our excellent new schedule is better than the confusing old one?"), and a double-barreled question asks two things at once ("How satisfied are you with the teachers and the bus service?"), so you can't tell which part a respondent is answering. The remedy for the second is to ask two questions. Answer choices should be mutually exclusive and cover all possibilities, which "1-2 hours," "2-3 hours," and "more than 3 hours" fail to do, since the ranges overlap and nothing fits less than an hour. A digital survey can also restrict answer formats with validation, checks on data types and ranges. A free-text age field collects "15," "fifteen," and "almost 16," whereas a field limited to whole numbers in a reasonable range keeps the data accurate and usable.

Web scrapers are programs that extract data from web pages. They gather what already exists, so the questions are about permission and fragility. Responsible use means checking a site's terms of service and its robots.txt file, a file in which a site's owner states limits on automated access, and limiting the rate of requests so that the site isn't overloaded. Visible doesn't mean permitted: a directory you can read in a browser may still be off limits to automated collection. The robots.txt file states those limits and doesn't enforce them, since it encrypts nothing and it is up to the scraper to respect it. Scrapers also break when a page's underlying structure changes, even if the page looks the same to human visitors. For these reasons, a documented API (an interface the site provides for programs) is preferable where one exists, because it offers a supported, structured form of the data.² An API still comes with terms, and it doesn't guarantee that the data is free of errors.

Protecting and judging what you have#

Once data has been collected, the collector becomes responsible for it. Data stewardship means protecting data in proportion to its sensitivity, limiting it to authorized people, de-identifying it where possible, and deleting it, according to a policy, when it is no longer needed. Proportion means that a club's health notes need stricter limits than its T-shirt sizes, even in the same data set. Limiting data means sharing only the fields a partner needs, with identifying information removed, since keeping data "just in case" carries risk with no benefit. De-identifying is harder than removing names. A roster stored beside "anonymous" responses defeats anonymity, and quasi-identifiers such as birth date and ZIP code can re-identify people when combined, especially in a small data set.

The last question to ask is whether the data is any good, and it has several parts, because data can succeed on one and fail on another. Accuracy asks whether the values are correct. Completeness asks whether the needed values are present, which is the problem if program records are missing for half of a district's schools. Consistency asks whether the same thing is always represented the same way, so "NY," "N.Y.," and "New York" are a consistency problem even though all three are accurate. Timeliness asks whether the data is current: housing prices that were correct five years ago are accurate as of that date and of little use to a buyer today. All four are judged against a purpose. A national census file may be excellent and still can't tell you how attendance changed week by week at one school, so excellent data can be wrong for your question.

Problems with consistency and completeness don't have to end a project, since many of them can be repaired, and repairing data is one of the jobs of the tools in the next section.

• data collection protocol: an advance plan for what is collected, from whom, how it is stored, who it is shared with, and why
• sampling bias: a sample that doesn't represent the population of interest
• sampling rate: how often a sensor records a measurement
• leading question: a question that pushes respondents toward an answer
• double-barreled question: a question that asks about two things at once
• validation: checking the type and range of what is entered
• web scraper: a program that extracts data from web pages
• robots.txt: a file stating a site owner's limits on automated access
• data stewardship: protecting data in proportion to its sensitivity, limiting access, de-identifying, and deleting when no longer needed
• quasi-identifiers: details such as birth date and ZIP code that can identify people when combined
• accuracy, completeness, consistency, timeliness: the dimensions of data quality: values are correct, present, uniformly represented, and current

Practice: practice questions for 0002.b

1. MWC1, Argument project: collecting and using data to make a claim. ↩

2. MWC1, Weather: dictionaries, web APIs, and distributed services. ↩