Start here Chapter 0

Target, Contribution, Uncertainty

Every weighted number answers three questions: who it describes, how each observation counts, and what could have come out differently.

DataATUS · synthetic population · interactive

Should I use the weights?

You have the file open. It came from IPUMS or from a CDC download page, and somewhere in it is a column called PERWT, or _LLCPWT, or WT06. The question you are asking is the one every applied researcher asks of a survey file and no econometrics sequence quite answers: should I use the weights, and which column is the weight? The software will not help you decide. Stata accepts [pw=] on any positive column; R's lm has a weights argument that means something else entirely; a referee who asks "did you use the survey weights?" will accept yes. The question feels like a yes-or-no about a column. It is three decisions about your estimate, and this chapter's job is to make you feel all three as yours before Chapter 1 starts settling them.

One file shows the three. In 2024, 442,292 adults picked up a telephone, agreed to a state health department interview, and answered one question: on how many of the past 30 days was your mental health not good? Count the answers, and 13.9% said 14 or more, the threshold CDC calls frequent mental distress. Apply the weight column that ships on the same file, and the figure is 15.6%. No answer changed. The 1.7 percentage points between the two numbers is the difference between describing the people who answered the phone and describing the adults they were called to stand for. That is the first decision, and it is the one "should I use the weights?" is usually about: how each observation counts.

The second decision is hidden in the word "adults." Which adults? Not the United States. Tennessee could not collect enough interviews to meet CDC's release standard and is absent from the file, and the three territories draw their samples differently and are set aside, so the weighted figure describes adults in 49 states and the District of Columbia, and only those who would answer the question (CDC 2024). Your paper's first table will say "U.S. adults" unless you decide otherwise, and the weights will not object. The third decision is how sure the number is, and it depends on how the software is told to read the same weight column. Read as a count of identical people, the column yields a standard error of 0.0023 percentage points; read as a precision, 0.055; read as a sampling weight with the survey's strata declared, 0.122. Same estimate, three intervals, the widest many times the narrowest, and only one of them describes the survey that was run.

So the question to ask of your file is not one but three, and the software asks none of them. It accepts a weight variable, and almost any column of positive numbers will do.

Target: who and what is my number about? Adults in 49 states, among those who would answer, in the case above. In the American Time Use Survey, the same diaries and the same weight give 3.56 hours of work per day across all person-days of people 15 and older and 8.27 hours on the days when employed people worked (BLS 2024). A different set of days is a different question, not a better or worse answer to the same one.
Contribution: how does each observation count? A weight gives each record the number of population units it stands for. Counting every BRFSS record once describes a sample in which older, college-educated adults are far more common than they are in the population; counting every ATUS diary once describes a year that is half weekends.
Uncertainty: what could have come out differently? Another draw of households, other people answering, another assignment of a policy. A standard error is a claim about one such repetition, and the analysis has to say which one. The three BRFSS standard errors are three different claims.

One question carries across the book, and it is the one to write at the top of your do-file:

Definition

The question to ask of any estimate

What is this number a weighted average of, and who chose the weights?

Nearly every estimate you will report is a weighted average of something: of respondents' answers, of cell means, of conditional effects. Sometimes a survey statistician chose the weights, knowing the selection probabilities. Sometimes you chose them on purpose. And sometimes the estimator chose them, and nobody noticed.

A worked example: one Ledger, filled in

A real survey never reveals the truth it estimates, which is why the argument about weights never ends. The first worked example uses a population in which the truth is known. This chapter's build script creates a synthetic population of 100,000 adults, a fifth of them rural, in which the share lacking a regular source of care is 14.1%. A stratified design samples 1,000 adults, half rural and half urban, so each rural adult is four times as likely to be selected as each urban adult. This is the design of every survey that oversamples a small group to say something about it.

Predict first

Rural adults in this population are more likely to lack a regular source of care. They are a fifth of the population and half of the sample. Will the unweighted sample share land above or below the population value? And will weighting each respondent by the inverse of their selection probability close the gap in every sample, or only on average?

In one simulated sample, the unweighted share is 18.8%. Weighting each rural respondent by 40 and each urban respondent by 160, the inverses of their selection probabilities, gives 12.2% (95% CI 10.2%–14.2%), with a design-based standard error of 1.0 percentage points. The weighted estimate misses the truth in this particular sample, and its interval still covers it. The unweighted estimate misses for a different reason: it describes a population that is half rural.

Repetition separates the two failures and answers the prediction. Across 2,000 repetitions of the same design, the unweighted share averages 20.1%, well above the truth; the weighted share averages 14.1% (Figure 0.1). Weighting closes the gap on average, not in every sample. The weighted share's spread across repetitions, 1.2 percentage points, is the quantity a design-based standard error estimates, and nominal 95 percent intervals built from the one-sample formula cover the truth in 94.7% of repetitions.

Two estimators under the same design, 2,000 repeated samples

Dot: average estimate across repetitions; bar: middle 95% of estimates

Truth
Weighted
Unweighted

Show data

Estimate for	Estimate	95% CI	n
Population truth	14.1%	—	—
Weighted mean (design weights)	14.1%	11.8%–16.5%	1,000
Unweighted mean	20.1%	17.7%–22.6%	1,000

Figure 0.1.Under the same design, the unweighted mean centers well above the population value while the weighted mean centers on it.Note: Rural adults (a fifth of the population) are sampled at four times the urban rate.Source: Synthetic population defined in ch00-start-here/build.py (seed 20260910); no survey data

Every headline result in the book comes with an Analysis Ledger like the one below: eight fields that answer the three questions in writing. Target population and estimand state the Target. Estimator, explicit weights, and implicit weights state the Contribution. Source of randomness and variance estimator state the Uncertainty. Assumptions state what must hold for the interpretation to stand. The Ledger is generated from the same artifact as the numbers it describes, so the two cannot drift apart. It is also a template: filled in for your own estimate, it is the paragraph a referee wants and rarely gets.

Analysis Ledger

A worked Ledger: share of a synthetic population lacking a regular source of care

Target population: The 100,000 adults of the chapter's synthetic population: a finite population fixed once by the build script's seed, a fifth of them rural.
Estimand: The population proportion lacking a regular source of care, a finite-population quantity that can be computed exactly here because the whole population is known.
Estimator: Hajek weighted mean over the 1,000 sampled adults: sum of w_i y_i divided by sum of w_i.
Explicit weights: Design weights d_i = 1/pi_i: 40 for rural adults (sampled 1 in 40) and 160 for urban adults (1 in 160). No nonresponse or calibration adjustment (g_i = 1), so w_i = d_i.
Implicit weights: None beyond the explicit weights: for a mean, each unit enters in proportion to w_i. Unweighted, rural adults would carry half the influence while making up a fifth of the population.
Source of randomness: Sampling only: which adults the stratified design selects. The population values are fixed.
Variance estimator: Stratified with-replacement (Taylor) variance with n - H = 998 df, checked against the spread of the weighted mean over 2,000 simulated repetitions of the same design.
Assumptions: Inclusion probabilities are known and positive; every sampled adult responds; the outcome is measured without error. Relax any of these and the weights stop being 1/pi_i.
Numbers reported: 14.1%
12.2%
18.8%
1.2percentage points
94.7%

Read the Ledger's assumptions field closely: known selection probabilities, complete response, accurate measurement. Real surveys relax all three. That is why their weights are more than $1/\pi_i$ , and why their variance estimators are more than a textbook formula. The BRFSS weight that opened the chapter is the product of five steps, only the first three of which are inverse probabilities; Chapter 2 takes it apart.

Three moves on one synthetic population

Before going back to your own file, spend a few minutes with a population in which nothing is hidden. The interactive below uses the book's canonical synthetic population, which the weight builder in Chapter 2 also uses: adults nested in neighborhoods, with an age and education structure, one group (called G) that is more often uninsured and concentrated in some neighborhoods, outcomes that vary from neighborhood to neighborhood, and a work-from-home effect on wages that differs across age-by-degree cells. Because the whole population is known, the widget can place the truth next to every estimator. Its three stages are the three decisions, one at a time: who enters the sample, how the sample is drawn, and what a regression averages. Make all three predictions before opening it.

Stage 1: who enters the sample

Predict first

Predict first: representation

The first stage makes members of group G more likely to be sampled, by a factor you choose, while the sample size stays fixed. Members of G are uninsured more often than everyone else. As the factor rises, predict what happens to the unweighted uninsured rate and to the weighted rate, in one sample and on average over repeated samples, measured against the population value.

Stage 2: how the sample is drawn

Predict first

Predict first: uncertainty

The second stage keeps the sample size and every person's selection probability fixed, and compares simple random samples of people with two-stage cluster samples: first a set of neighborhoods, then the same number of residents in each, from many neighborhoods with a few residents apiece to a few neighborhoods with many. Predict what happens to the center of the sampling distribution, to its width, to the naive standard error that treats people as independent draws, and to a standard error clustered on neighborhoods.

Stage 3: what a regression averages

Predict first

Predict first: the regression target

The third stage regresses log wages on working from home, with dummies for the six age-by-degree cells. Two sliders set how much the effect of working from home differs across cells and how unequal the treated shares are. Predict whether the regression's target lands closer to the population average treatment effect or to the average effect among the treated, and which cells receive the most implicit weight.

One population, three questions

A synthetic population of 20,000 adults in 200 neighbourhoods (version 1.0.0, population seed 20260910). Each stage changes one thing.

Stage

1. Who enters the sample2. How the sample was drawn3. What the regression targets

How much more likely a member of group G is to be sampled4×

Selection probability: 13.8% for G, 3.4% for everyone else. Design weight d = 1/π.

Seed 1n = 1,000

Population

G 15%

not G

This sample

G 42%

not G

Across 200 samples at this setting the unweighted mean averages 16.5% and the weighted mean 12.3%, against a truth of 12.4%. Members of G are uninsured more often (26% against 10%), so a sample that over-represents them overstates the rate unless each person counts 1/π times.

What is held fixed. The population, the outcome (uninsured), and the sample size. Only who is likely to enter the sample changes; the design weights d = 1/π undo exactly that.

What the three stages show

In the first stage only the unweighted rate moves with the oversampling factor, because only it describes the sample's composition: it overstates the uninsured rate more as G is overrepresented. The weighted rate, in which each person counts $1/\pi_i$ times, stays centered on the population value across repeated samples, though any single sample can miss. The repair costs precision, which Chapter 2 measures with Kish's approximation.

In the second stage the center of the distribution stays put while its width grows as fewer neighborhoods supply more residents each, and the naive standard error does not notice. A standard error clustered on neighborhoods tracks the actual spread on average. It runs slightly high when half the neighborhoods are sampled, because it ignores the finite-population correction, and with only a handful of neighborhoods it rests on a handful of cluster totals.

In the third stage the regression's target is a weighted average of cell effects, with weights proportional to each cell's share times $p(1-p)$ for its treated share $p$ . Set either slider to zero and the target coincides with the population average effect: equal treated shares make the weights proportional to population shares, and equal effects make every average the same. Otherwise the target sits wherever the weights put it, and cells where treatment is split most evenly count for more than their share of the population. Nobody chose those weights. That is the third decision made by default, and Chapters 4 and 5 are about noticing it.

The vocabulary, in brief

Each term below is developed in the chapter where it does its work, and Appendix A collects the notation. None of it has to be learned before reading on.

Estimand, estimator, estimate

The estimand is the population quantity you want, stated without reference to any data: the share of US adults who currently smoke, in 2024. The estimator is the rule that turns a sample into a number: a weighted proportion over respondents. The estimate is what the rule returns on the sample in hand. Arguments about "the right answer" are often arguments about estimands in disguise (Lundberg, Johnson, and Stewart 2021).

Five populations

Every gap between these is a place where an analysis can drift, and Chapter 1 follows all five through one estimate.

The target population is the one the result is meant to describe.
The frame is what the survey actually draws from.
The sampled population is the part of the frame with a positive chance of selection.
The respondent population is those in the sampled population who would answer if selected.
The analytic population is who remains after the analyst's own restrictions and missing-data rules.

The sample, the units actually selected, and the respondents, the selected units that answered, are what one survey realizes from the sampled and respondent populations. They are not populations of their own.

Kinds of weight

A survey weight is not one thing, and the book gives each part its own symbol.

The design weight $d_i = 1/\pi_i$ undoes unequal selection probabilities $\pi_i$ .
A nonresponse and calibration adjustment $g_i$ moves the weights so that respondents match known population totals, and the final weight shipped on the file is $w_i = d_i g_i$ . BRFSS builds its weight in this order: a design weight from the telephone stratum, the number of adults, and the number of phones; a correction for people reachable on both a landline and a cell phone; truncation of extreme design weights; then raking to demographic margins, state by state (CDC 2024). After raking, the weight is no longer an inverse selection probability (Gelman 2007).
A frequency weight $f_i$ says a row stands for $f_i$ identical observations. A precision weight $a_i$ says a row is more or less noisy than the others, as when each row is a group mean. Read as a frequency, precision, or sampling weight, the same column gives the same point estimate and three different standard errors, which is where the three BRFSS intervals above came from (Chapter 2).
A regression places implicit weights $\omega_i$ on units or cells that nobody supplied. With a binary treatment and a full set of dummies for covariate cells, ordinary least squares averages the cell-specific contrasts with weights proportional to the cell's share of the data times $p_x(1-p_x)$ , where $p_x$ is the cell's treated share (Angrist 1998). Cells in which nearly everyone, or nearly no one, is treated barely count. Equal case weights do not imply equal weighting of heterogeneous conditional effects (Aronow and Samii 2016; Słoczyński 2022).
Standardization weights $s_a$ redefine the comparison population: an age-standardized prevalence reweights age groups to a fixed standard population so that populations with different age structures can be compared (Chapter 3).

Four repetitions

A standard error is a statement about a hypothetical repetition, and the analysis has to say which one.

Sampling: redraw the sample under the same design. NHANES draws counties, and its public file supports only 15 design degrees of freedom for the August 2021–August 2023 cycle; BRFSS treats every respondent as a separate sampling unit, which leaves it 455,277 design degrees of freedom in 2024. Both are called complex surveys, and their uncertainty comes from opposite places (Chapter 3).
Assignment: rerun the process that decided who was treated. When a policy varies across states, the number of independent assignments is the number of states, however many records there are (Chapter 6; (Abadie et al. 2023)).
Response: repeat the process that decided who answered. Because nonresponse and calibration adjustments are estimated from the sample, the weights themselves vary from sample to sample.
Model: treat the finite population as one draw from a superpopulation. When every unit is observed, sampling uncertainty disappears, but assignment and model uncertainty can remain (Abadie et al. 2020).

The slogan that "weights get the point estimate and the design gets the uncertainty" survives only with a correction. The weights help define the estimator. The sampling design and the procedure that built the weights together determine how that estimator varies over repeated samples. Replicate weights that rerun the weighting in every replicate are one way producers package that fact for users.

Box

Four effective samples that are not the same thing

"Effective sample size" names at least four different objects. They answer different questions, and none converts into another.

Kish's effective sample size prices the variance cost of unequal weights for a mean, under restrictive conditions. It says nothing about bias (Chapter 2).
The design effective sample size prices everything the design does to one estimator, so a survey has no single one (Chapter 3).
The regression-effective sample is a composition, not a count: it says whose effects a regression coefficient summarizes (Chapter 5).
Meng's data-defect effective sample size prices bias as well as variance when inclusion is correlated with the outcome, and can be tiny for an enormous sample (Meng 2018).

Formulas and conditions

$\nKish = n/(1+\CV(w)^2) = (\sum_i w_i)^2/\sum_i w_i^2$ , under an outcome unrelated to the weights, no clustering, and weights treated as fixed.
$\ndesign(\hat\theta) = n/\DEFF(\hat\theta)$ , where the design effect $\DEFF(\hat\theta)$ is the design-based variance of one estimator divided by its variance under simple random sampling of the same size. Because it depends on the estimator and the outcome, one dataset yields several.
The sample reweighted by the implicit weights $\omega_i$ a regression places on each unit's conditional effect; with a binary treatment and correctly specified controls, $\omega_i = (D_i - \E[D_i \mid X_i])^2$ . With linear controls and a treatment probability that is not linear in them, some implicit weights can be negative.
$\nDDC$ is the size of a simple random sample with the same mean squared error as a large sample whose inclusion is correlated with the outcome. It belongs to a planned essay, When weighting is not enough.

The first two are variance ratios, the third is a distribution over units, and the fourth is an equivalence in mean squared error. Appendix B derives each with its conditions.

One object, three dialects

The same objects travel under different names in different fields, which is one reason your advisor and your referee can both be right. This is a short version of the crosswalk in Appendix A.

The object	Survey statistics	Econometrics	Causal inference
The quantity sought	a finite-population parameter: total, mean, ratio	a population parameter, often a linear-projection coefficient	an estimand: PATE, ATT, CATE
How a unit enters	sampling weight, adjusted and calibrated	probability or analytic weight	inverse-propensity or balancing weight; implicit regression weight
Units that move together	primary sampling units within strata	clusters	units assigned together
The repetition	redrawing the sample under the design	redrawing from the data-generating process	rerunning the assignment
The variance	Taylor linearization or replicate weights	sandwich and cluster-robust estimators	randomization-based variance

Reader routes

Describing a population (a prevalence, a rate, a mean, a gap between groups): Chapter 1 on estimands, Chapter 2 on where weights come from, Chapter 3 on design-based uncertainty, and Lab 3 on replicate weights.
Estimating a regression coefficient or an effect: Chapter 4 on explicit weighting, Chapter 5 on implicit weights under heterogeneous effects, and Chapters 6 and 7 on clustering as a question about sampling and assignment design.
Declaring a specific survey: Chapter 8, with release guides for six surveys and declarations in Python, R, and Stata.
Checking a symbol, a derivation, or a data source: Appendix A for notation and the crosswalk, Appendix B for derivations with their conditions, and Appendix C for data, code, and validation.

Before Chapter 1

Each chapter that follows opens with a question a researcher actually asks of a file, one that has at least two defensible-looking answers, and each isolates what changes the answer: the target, the representation weights, the design, the explicit weighting of a regression, the implicit weights inside it, or the number of independent units. Chapter 1 begins with the decision that has to be settled before any column is touched: what the number is supposed to describe.