Part II Chapter 6

Sampling Design, Assignment Design, and Clustering

Millions of survey records, dozens of policy shocks: clustering as a question about sampling and assignment.

DataACS 2011–2019 · Medicaid expansion · few-cluster inference

Thirty states, one decision each

Your regression has a policy that varies by state and an outcome measured on people, and its standard error rests on an argument you have not yet made. The software, left alone, treated every record as an independent draw. Your advisor says cluster by state. The referee says the policy was administered by counties, so cluster by county. One methods paper says the level of treatment assignment, another the level of sampling. Each choice gives a different interval around the same coefficient, and the widest is several times the narrowest. "The widest, to be safe" is not an answer, because the width of an interval is not evidence that it describes anything. The question that decides is what would change if the study were run again, and that question has an answer that can be read off the design. This chapter reads it off one study.

On January 1, 2014, the Affordable Care Act's expansion of Medicaid to adults with incomes up to 138 percent of the federal poverty line took effect wherever a state had agreed to it. The Supreme Court's 2012 ruling had made the expansion a state's choice, and states chose differently. Arkansas, Kentucky, and West Virginia expanded on the first day, and so did Arizona, Colorado, Hawaii, Illinois, Iowa, Maryland, Nevada, New Mexico, North Dakota, Ohio, Oregon, and Rhode Island. Texas, Florida, Georgia, North Carolina, Tennessee, and ten other states declined, and had still not expanded by the end of 2019. Between the two groups sit states that had already covered low-income adults before 2014, California and New York among them, and states such as Michigan, Pennsylvania, and Louisiana that expanded later.

The American Community Survey (ACS) interviews people in every one of these states every year and asks each whether they have health coverage on the day of the interview (Ruggles et al. 2025). For the 15 first-day expanders and 15 states that never expanded in the window, it holds 1,484,005 records of adults ages 19 to 64 in households at or below 138 percent of poverty, 2011 through 2019. A PERWT-weighted linear probability model of being uninsured, with state and year fixed effects and an indicator for expansion states from 2014 on, puts the change in the uninsured-rate gap at −8.0 percentage points. Treat every record as an independent draw, with a heteroskedasticity-robust (HC1) standard error, and the coefficient comes back as −8.0 (95% CI −8.4 to −7.5). Change one argument so that the standard error is clustered by state, and the same coefficient comes back as −8.0 (95% CI −10.7 to −5.3), with a standard error 6.1× as large.

1,484,005

person records in the design

states, each of which made one policy decision

Which interval describes the uncertainty? Two things could change if the study were run again. The Census Bureau could draw a different sample of households, and the legislatures and governors of these 30 states could have decided differently. More interviews help measure each state's outcome. They do not create more state decisions. Nor was expansion randomized: states chose, and the coefficient has a causal reading only under the comparison assumptions stated below. Every interval in this chapter is conditional on those assumptions and on the rules that define the cohort.

Uninsured share, low-income adults 19–64

PERWT-weighted; bands are 95% SDR replicate intervals (survey sampling only)

January 2014 expansion cohort (15 states)
Not expanded by 2019 (15 states)

Show data

Series	ACS survey year	Uninsured share	95% CI
January 2014 expansion cohort (15 states)	2011	39%	38%–39%
January 2014 expansion cohort (15 states)	2012	38%	37%–39%
January 2014 expansion cohort (15 states)	2013	37%	36%–37%
January 2014 expansion cohort (15 states)	2014	26%	26%–26%
January 2014 expansion cohort (15 states)	2015	19%	19%–20%
January 2014 expansion cohort (15 states)	2016	17%	17%–18%
January 2014 expansion cohort (15 states)	2017	17%	17%–17%
January 2014 expansion cohort (15 states)	2018	18%	17%–18%
January 2014 expansion cohort (15 states)	2019	18%	17%–18%
Not expanded by 2019 (15 states)	2011	49%	49%–50%
Not expanded by 2019 (15 states)	2012	48%	47%–48%
Not expanded by 2019 (15 states)	2013	47%	46%–47%
Not expanded by 2019 (15 states)	2014	41%	41%–42%
Not expanded by 2019 (15 states)	2015	37%	37%–37%
Not expanded by 2019 (15 states)	2016	36%	35%–36%
Not expanded by 2019 (15 states)	2017	36%	36%–37%
Not expanded by 2019 (15 states)	2018	36%	36%–37%
Not expanded by 2019 (15 states)	2019	37%	37%–37%

Figure 6.1.Uninsurance fell in both groups from 2014 and much further in the expansion cohort; the gap between the groups was stable before 2014 and wider after. Bands show survey sampling uncertainty only.Note: Adults 19–64 with family income at or below 138% of poverty; household population (POVERTY excludes group quarters).Source: IPUMS USA, ACS 1-year samples 2011–2019 (analysis/usa/acs/part_2010_2014.parquet, part_2015_2019.parquet); expansion dates in _data/expansion_dates.csv (KFF; medicaid.gov); replicate weights REPWTP1–80 (analysis/usa/acs_repwt)

The gap between the groups in Figure 6.1, cohort minus comparison, was −10.3 percentage points in 2011 and −9.9 in 2013. It was −15.1 in 2014 and −19.2 in 2019.

One cohort, one comparison, one window

Expansion did not arrive on one date. Michigan and New Hampshire expanded later in 2014; Pennsylvania, Indiana, and Alaska in 2015; Montana and Louisiana in 2016; Virginia and Maine in 2019. A single two-way fixed effects coefficient estimated across all of these dates averages many two-group comparisons, some of which use already-expanded states as controls for later ones and receive negative weight when effects change over time (Goodman-Bacon 2021); estimators designed for staggered adoption avoid this (Callaway and Sant'Anna 2021). This chapter avoids it by construction: one adoption date, and a comparison group that does not adopt within the window, so every comparison is between expansion states and never-expanded states, 2014–2019 against 2011–2013. That removes the staggered-adoption problem. It does not make the comparison causal, which rests on the assumptions below.

The rules sort the 51 state units, the 50 states and the District of Columbia, using effective coverage dates from KFF's tracker of state decisions, checked state by state against federal and state documents (KFF 2026). Of the 2,578,630 records in the full 2011–2019 file, the design keeps 1,484,005. Figure 6.2 shows the result.

Cohort. ACA expansion effective January 1, 2014. Of the 25 units that meet this, 10 are set aside because 2014 was a smaller change for them and 2011–2013 is not a clean before: six had already extended Medicaid to low-income adults statewide in 2010 or 2011 through the ACA's early option or a Section 1115 waiver (California, Connecticut, the District of Columbia, Minnesota, New Jersey, Washington) (Sommers et al. 2013), and five already covered low-income parents and childless adults broadly (Delaware, the District of Columbia, Massachusetts, New York, Vermont) (Kaestner et al. 2017; Miller, Johnson, and Wherry 2021).
Comparison. No ACA expansion in effect by December 31, 2019, and no statewide change in adult eligibility inside the window. Wisconsin and Utah fail the second test: in April 2014 a Wisconsin waiver opened coverage to childless adults up to the poverty line while lowering parents' limit to it, and in April 2019 Utah began a partial expansion.
Later adopters. Set aside.
Window. ACS survey years 2011 through 2019: three years before and six after. The window opens after the 2008–2009 recession and after the ACA's 2010 dependent-coverage provision reached young adults, and it closes before the 2020 ACS, an experimental release collected during the pandemic.

Small programs that stay in

Colorado's capped program for the poorest childless adults, county pilots in Cook County (Illinois), Cuyahoga County (Ohio), and St. Louis (Missouri), and eligibility cuts in Arizona, Hawaii, and Illinois are not counted as expansions. Dropping the 6 design states with any of them gives −10.3; the closing panel returns to this.

Who is in the design

Rules applied to the verified expansion-date panel (_data/expansion_dates.csv)

Table of the five design groups with the rule that defines each, the states in it, the number of states, and ACS person records 2011–2019.
Group	Rule	States	States (n)	Records
January 2014 expansion cohort (treated)	ACA expansion effective 2014-01-01; no statewide expansion for low-income adults in 2010–2011 and no broad adult coverage before 2014	AR, AZ, CO, HI, IA, IL, KY, MD, ND, NM, NV, OH, OR, RI, WV	15	543,835
Comparison	No ACA expansion in effect by 2019-12-31 and no statewide change in adult eligibility during 2011–2019	AL, FL, GA, ID, KS, MO, MS, NC, NE, OK, SC, SD, TN, TX, WY	15	940,170
Set aside: January 2014 expanders with earlier coverage	Expanded 2014-01-01 but already covered low-income adults statewide (ACA early option or Section 1115 waiver, 2010–2011) or broadly (pre-ACA programs)	CA, CT, DC, DE, MA, MN, NJ, NY, VT, WA	10	682,937
Set aside: later adopters through 2019	ACA expansion effective after 2014-01-01 and by 2019-12-31	AK, IN, LA, ME, MI, MT, NH, PA, VA	9	356,912
Set aside: other eligibility change in the window	No ACA expansion by 2019, but a statewide waiver changed adult eligibility during 2011–2019	UT, WI	2	54,776

Figure 6.2.The cohort and comparison groups are what remains after every state with an earlier, partial, or later change is set aside for a stated reason.Note: Dates are effective (coverage) dates, not adoption or approval dates.Source: KFF Status of State Medicaid Expansion Decisions; medicaid.gov; see _data/expansion_dates.csv for per-state sources

The rules shape the answer, not only its interval. Adding back the 10 January 2014 states with earlier coverage gives −6.9, most of a state-clustered standard error (1.32 points) away; the stricter rule in the note above gives −10.3, more than one away. Every interval here describes uncertainty given the design, none uncertainty about it.

Caution

What the design assumes

The comparison has a causal reading only if, without expansion, uninsurance in the cohort would have moved in parallel with the comparison states, if no state changed course in anticipation, and if one state's policy did not change coverage in another. This chapter takes those assumptions as given. Its subject is inference conditional on the design: given this comparison, how uncertain is the number, and why?

The 2×2 comparison

PERWT-pooled uninsured shares by group and period

Two-by-two table of uninsured shares before (2011–2013) and after (2014–2019) for the expansion cohort and the comparison states, with each group's change and the difference in differences.
Group	2011–2013	2014–2019	Change (points)
January 2014 expansion cohort (15 states)	37.9%	19.3%	−18.6
Not expanded by 2019 (15 states)	47.8%	37.2%	−10.6
Difference in differences	—	—	−8.0

Figure 6.3.Uninsurance fell in both groups between the two periods, and further in the expansion cohort.Note: Pooled means weight each state by its population in each period; the regression with state fixed effects compares each state with itself, so its coefficient differs slightly.Source: IPUMS USA, ACS 1-year samples 2011–2019 (analysis/usa/acs/part_2010_2014.parquet, part_2015_2019.parquet); expansion dates in _data/expansion_dates.csv (KFF; medicaid.gov)

Pooling 2011–2013 and 2014–2019 (Figure 6.3), the cohort's uninsured share changed by −18.6 percentage points and the comparison group's by −10.6, a difference in differences of −8.0. The regression with state and year fixed effects compares each state with itself and gives −8.0; it weights each state-year cell by its population, so the two need not agree, although here they do to the displayed precision. The year-by-year version, the event study, replaces the single post-2014 indicator with one indicator per year, each measured against 2013 (Figure 6.4).

Year-by-year gap relative to 2013

Expansion cohort minus comparison, with state and year fixed effects; 95% intervals clustered by state, t(29)

Show data

Series	ACS survey year	Difference in uninsured share (percentage points)	95% CI
Gap relative to 2013 (state-clustered 95% interval)	2011	−0.3	−2.4–1.9
Gap relative to 2013 (state-clustered 95% interval)	2012	0.5	−0.7–1.7
Gap relative to 2013 (state-clustered 95% interval)	2013	0.0	—
Gap relative to 2013 (state-clustered 95% interval)	2014	−5.2	−7.7–−2.7
Gap relative to 2013 (state-clustered 95% interval)	2015	−7.9	−10.8–−4.9
Gap relative to 2013 (state-clustered 95% interval)	2016	−8.3	−11.5–−5.0
Gap relative to 2013 (state-clustered 95% interval)	2017	−9.1	−12.1–−6.1
Gap relative to 2013 (state-clustered 95% interval)	2018	−8.3	−11.8–−4.8
Gap relative to 2013 (state-clustered 95% interval)	2019	−9.1	−12.3–−6.0

Figure 6.4.The year-by-year gap is near zero before expansion and grows after it; the state-clustered intervals are several percentage points wide.Note: 2013 is the reference year (no interval). Identification by parallel trends is assumed, not established.Source: IPUMS USA, ACS 1-year samples 2011–2019 (analysis/usa/acs/part_2010_2014.parquet, part_2015_2019.parquet); expansion dates in _data/expansion_dates.csv (KFF; medicaid.gov)

The 2011 and 2012 coefficients are −0.3 and 0.5 percentage points, with a joint p-value of 0.31 under state clustering. After expansion they move from −5.2 in 2014 to −9.1 in 2019. Small pre-period coefficients are what parallel trends predicts, but they are not evidence that it holds: such tests have little power against the violations that matter, and conditioning an analysis on passing one alters the properties of what follows (Roth 2022).

Stage one: what the survey knows

The analysis has two stages, and they line up with the two things that could change in a repetition.

The first stage estimates the uninsured share in each state and year. The ACS selects addresses within geographic strata and interviews everyone at a selected address. The person weight, PERWT, turns each record into a count of people. The 80 replicate weights repeat the sampling and weighting in a structured way (successive difference replication), so the spread of an estimate across replicates, scaled by 4/80, estimates its sampling variance with 79 degrees of freedom, including the uncertainty that weight construction adds (Fay and Train 1995; U.S. Census Bureau 2024). For the 270 state-year cells in the design, the median replicate standard error is 1.0 percentage points.

Predict first

The opening regression used 1,484,005 records. How many distinct numbers does its coefficient depend on?

The second stage is the policy comparison, and it needs only the first stage's output. Write $\bar y_{st}$ for the PERWT-weighted uninsured share in state $s$ and year $t$ , and $W_{st}$ for the cell's total weight. The state effect $\alpha_s$ , the year effect $\lambda_t$ , and the treatment indicator $D_{st}$ are all constant within a cell, so the weighted sum of squared residuals splits in two:

\begin{aligned} \sum_{s,t}\sum_{i\in(s,t)} w_i\bigl(y_i-\alpha_s-\lambda_t-\beta D_{st}\bigr)^2 &=\sum_{s,t}\sum_{i\in(s,t)} w_i\bigl(y_i-\bar y_{st}\bigr)^2 \\ &\quad+\sum_{s,t} W_{st}\bigl(\bar y_{st}-\alpha_s-\lambda_t-\beta D_{st}\bigr)^2 . \end{aligned}

The first term does not involve the parameters. Minimizing over the person records therefore gives exactly the coefficient from a regression of the 270 cell shares on the same regressors, weighted by $W_{st}$ ; the build script confirms that the two agree to machine precision. A record affects the coefficient only through its cell's weighted share.

The identity concerns the point estimate. Whether the cell shares are independent pieces of information is the design question, and Figure 6.5 shows why the answer is not obvious.

Each state's change, with its survey interval

Change in the uninsured share from 2011–2013 to 2014–2019; 95% SDR replicate intervals

Treated
Control

Show data

Group	Estimate for	Estimate	95% CI	n
January 2014 expansion cohort	Kentucky	−26.1	−27.2–−25.0	45,512
January 2014 expansion cohort	West Virginia	−25.6	−27.2–−24.1	20,036
January 2014 expansion cohort	Nevada	−23.3	−25.4–−21.2	24,149
January 2014 expansion cohort	New Mexico	−23.1	−25.0–−21.2	23,929
January 2014 expansion cohort	Arkansas	−23.1	−24.7–−21.5	32,393
January 2014 expansion cohort	Oregon	−21.8	−23.2–−20.4	33,965
January 2014 expansion cohort	Colorado	−18.7	−20.2–−17.3	36,534
January 2014 expansion cohort	Rhode Island	−18.4	−21.0–−15.8	6,932
January 2014 expansion cohort	Illinois	−17.3	−18.3–−16.4	94,452
January 2014 expansion cohort	Ohio	−16.8	−17.6–−16.0	97,474
January 2014 expansion cohort	Arizona	−14.9	−16.0–−13.8	65,643
January 2014 expansion cohort	Iowa	−14.2	−16.0–−12.4	19,651
January 2014 expansion cohort	Maryland	−11.6	−12.9–−10.3	30,856
January 2014 expansion cohort	Hawaii	−8.9	−11.5–−6.4	8,027
January 2014 expansion cohort	North Dakota	−6.7	−10.9–−2.4	4,282
Not expanded by 2019	Florida	−13.9	−14.7–−13.2	171,063
Not expanded by 2019	North Carolina	−11.4	−12.1–−10.6	92,075
Not expanded by 2019	Kansas	−11.3	−13.5–−9.0	20,866
Not expanded by 2019	Idaho	−11.1	−13.3–−8.8	14,120
Not expanded by 2019	South Carolina	−11.0	−12.1–−9.9	46,572
Not expanded by 2019	Georgia	−10.4	−11.2–−9.6	96,609
Not expanded by 2019	Tennessee	−10.3	−11.4–−9.2	64,299
Not expanded by 2019	Texas	−9.4	−10.0–−8.9	235,004
Not expanded by 2019	Missouri	−9.2	−10.3–−8.1	53,042
Not expanded by 2019	Alabama	−8.8	−10.0–−7.5	49,924
Not expanded by 2019	Mississippi	−8.7	−10.2–−7.1	36,331
Not expanded by 2019	Oklahoma	−8.6	−10.2–−6.9	38,079
Not expanded by 2019	Wyoming	−7.8	−12.0–−3.6	3,952
Not expanded by 2019	Nebraska	−7.5	−10.2–−4.7	12,120
Not expanded by 2019	South Dakota	−6.2	−9.8–−2.6	6,114

Figure 6.5.The survey measures each state's change precisely; the expansion states differ from one another by far more than their survey intervals.Note: Replicate r is used across all pooled years; the years are independent samples, so cross-year replicate terms average to zero.Source: IPUMS USA, ACS 1-year samples 2011–2019 (analysis/usa/acs/part_2010_2014.parquet, part_2015_2019.parquet); expansion dates in _data/expansion_dates.csv (KFF; medicaid.gov); replicate weights REPWTP1–80 (analysis/usa/acs_repwt)

Each state's change from 2011–2013 to 2014–2019 has a median replicate standard error of 0.8 percentage points. Across the cohort states the changes have a standard deviation of 6.0 percentage points; across the comparison states, 1.9. Survey noise is not what limits the policy comparison. The limit is how differently states moved, and how few states there are to average over.

Clustering is a question about the design

Clustering is usually taught as a rule about errors: if residuals are correlated within groups, cluster by group. The rule cannot say which groups. Uninsurance residuals are correlated within states, and also within age groups, occupations, and genders. Abadie et al. (2023) replace the rule with a question about the design: what mechanism, if the study were repeated, would move many observations together? There are two candidates.

Sampling design. When a sample is drawn in clusters, a new sample replaces whole clusters at once, so the observations in a cluster rise and fall together across repetitions. The ACS household is such a cluster: the survey selects an address and records everyone there. The fraction of clusters sampled matters. The ACS sees a small fraction of the nation's households, the setting where conventional clustering on the sampling unit fits; when most of a population's clusters are in the sample, sampling alone gives little reason to cluster, and conventional clustered standard errors can be severely inflated (Abadie et al. 2023).

Assignment design. When treatment is assigned in clusters, a different assignment changes the treatment of everyone in a cluster at once. Expansion was assigned to states: every low-income adult in Kentucky lived under Kentucky's decision, and every one in Tennessee under Tennessee's.

Neither mechanism operates by gender. The ACS does not sample genders, and no state expanded Medicaid for women only, so the argument clusters by state and not by gender even though residuals are correlated within both. For the same reason a completely randomized experiment needs no clustering: if eligibility were assigned person by person in an independently drawn sample, nothing would move a state's records together, however correlated their outcomes.

The full statement has four parts: identify every sampling or assignment mechanism that induces common uncertainty; in a nested hierarchy, let the coarsest relevant level govern conventional clustering; treat mechanisms that do not nest, such as a sample drawn by metropolitan area and a policy assigned by state, with multiway or other methods; and let the sampled fraction of clusters and the inferential target both matter. The familiar advice to "cluster at the coarser level" is the second part, not the whole statement.

Here the mechanisms nest: persons within households, households within state-years, state-years within states. Which is relevant depends on the target. For a descriptive target, the change in the gap between these particular states with their policies taken as fixed, only the sample varies, and the household is the level. For the policy target, the effect of expansion, the assignment varies too, and the state is the coarsest relevant level. Every state that meets the rules is observed, so the case for clustering by state is an assignment argument, not a sampling one; design-based uncertainty of this kind remains even when the sample is the whole population (Abadie et al. 2020). State fixed effects remove level differences between states, not shocks that persist within a state from year to year, which is why clustering below the state understates uncertainty in panel difference-in-differences (Bertrand, Duflo, and Mullainathan 2004).

Predict first

The state-clustered variance is a sum over clusters. With 1,484,005 records in 30 states, how many terms does the sum have?

Let $\tilde D_{st}$ be the treatment indicator after the weighted removal of state and year effects, and $\hat e_{st}$ the cell residual. The cluster-robust (CRV1) variance of the coefficient, with states as clusters $g$ , is

\widehat{\Var}_{\mathrm{CRV1}}(\hat\beta)=\frac{G}{G-1}\,\frac{N-1}{N-K}\, \frac{\sum_{g=1}^{G}\Bigl(\sum_{t} W_{gt}\,\tilde D_{gt}\,\hat e_{gt}\Bigr)^{2}}{\Bigl(\sum_{s,t} W_{st}\,\tilde D_{st}^{2}\Bigr)^{2}} ,

where $G$ is the number of states, $N$ the number of records, and $K$ the number of parameters not absorbed within clusters. The numerator has one term per state however many records each state contributes, and inference uses a $t$ distribution with $G-1$ degrees of freedom. The records determine how precisely each term is measured; the states determine how many terms there are.

The same coefficient therefore carries two ledgers, one for each target. Read them side by side: the estimator and both kinds of weight are the same, while the estimand, the source of randomness, the variance estimator, and the assumptions change.

Analysis Ledger

The same number as a survey description of fixed states

Target population: civilian and military household population ages 19–64 with family income at or below 138% of the poverty threshold (IPUMS POVERTY 1–138), in the 30 design states, 2011–2019, with the states and their policies taken as given
Estimand: How much more the uninsured share fell in the cohort states than in the comparison states, as a finite-population descriptive contrast (no causal claim)
Estimator: Same regression and same coefficient as the policy ledger
Explicit weights: PERWT
Implicit weights: Same as the policy ledger
Source of randomness: ACS sampling of addresses (households) within strata; nothing about which states expanded is treated as random
Variance estimator: Successive difference replication with the 80 ACS person replicate weights, scale 4/80, df 79; CRV1 by household as an approximation
Assumptions: The ACS replicate weights represent the sampling and weighting variability; the descriptive target makes no counterfactual claim
Numbers reported: −8.0percentage points
−8.0percentage points
−8.0percentage points

Analysis Ledger

Medicaid expansion and uninsurance: inference for the policy comparison

Target population: civilian and military household population ages 19–64 with family income at or below 138% of the poverty threshold (IPUMS POVERTY 1–138), in the 30 design states (January 2014 expansion cohort and states not expanded by 2019), 2011–2019
Estimand: Average effect of the 2014 expansion on the uninsured share among people in the expansion cohort, 2014–2019 (an ATT), under parallel trends
Estimator: PERWT-weighted linear probability model with state and year fixed effects; equivalently WLS on state-year means weighted by population
Explicit weights: PERWT (ACS person weight) to represent each state-year population; cells enter stage two in proportion to their weighted population
Implicit weights: Regression weights across states and years: each state's contribution depends on its population and on its within-state variation in treatment; larger states count more
Source of randomness: Assignment of expansion across states (the policy was chosen state by state), plus ACS sampling of households within each state-year
Variance estimator: CRV1 clustered by state with t(29); CRV3 jackknife and wild cluster restricted bootstrap as few-cluster checks; randomization inference over assignment as a sensitivity
Assumptions: Parallel trends between cohort and comparison (assumed, not established); no anticipation; no spillovers across states; states' shocks independent of one another; the stated cohort and exclusion rules
Numbers reported: −8.0percentage points
−8.0percentage points
−8.0percentage points
−8.0percentage points
−8.0percentage points

The standard-error ladder

Predict first

Eight variance rules follow, all applied to the same coefficient. Before looking, predict one of them. Clustering by state-year allows for shocks shared within a state in a given year. Will its interval land nearer the record-level interval or the state-clustered one?

One coefficient, eight intervals

95% intervals for the expansion coefficient under each variance rule

Naive
Design-based

Show data

Group	Estimate for	Estimate	95% CI
Neither: no mechanism	HC1: person records independent	−8.0	−8.4–−7.5
Sampling design	CRV1: ACS household clusters	−8.0	−8.5–−7.5
Sampling design	SDR: 80 ACS replicate weights	−8.0	−8.4–−7.6
Neither: a level with no mechanism	CRV1: state-year cells	−8.0	−9.2–−6.7
Assignment design	CRV1: states	−8.0	−10.7–−5.3
Assignment design	CRV3 jackknife: states	−8.0	−11.0–−5.0
Assignment design	Wild cluster bootstrap: states	−8.0	−11.2–−5.2
Sensitivity (assignment as if random)	Randomization inference over assignment	−8.0	−10.8–−5.4

Figure 6.6.Intervals that treat records, households, or state-years as independent are narrow; the four state-level intervals are several times wider and close to one another.Source: IPUMS USA, ACS 1-year samples 2011–2019 (analysis/usa/acs/part_2010_2014.parquet, part_2015_2019.parquet); expansion dates in _data/expansion_dates.csv (KFF; medicaid.gov); replicate weights REPWTP1–80 (analysis/usa/acs_repwt)

The standard-error ladder

Same coefficient in every row; the columns say what each rule treats as an independent draw

Table of eight variance rules for the same expansion coefficient, listing what each treats as independent, the number of clusters, the standard error, its ratio to HC1, and the 95% interval.
Variance rule	Independent units	Mechanism it represents	Clusters	SE (points)	SE / HC1	95% interval (points)	df
HC1: person records independent	Every record	Neither: no mechanism	—	0.22	1.00×	−8.4 to −7.5	large
CRV1: ACS household clusters	Households	Sampling design	964,784	0.26	1.19×	−8.5 to −7.5	964,783
SDR: 80 ACS replicate weights	Households, within ACS strata	Sampling design	—	0.19	0.89×	−8.4 to −7.6	79
CRV1: state-year cells	State-years	Neither: a level with no mechanism	270	0.63	2.91×	−9.2 to −6.7	269
CRV1: states	States	Assignment design	30	1.32	6.06×	−10.7 to −5.3	29
CRV3 jackknife: states	States	Assignment design	30	1.46	6.70×	−11.0 to −5.0	29
Wild cluster bootstrap: states	States	Assignment design	30	—	—	−11.2 to −5.2	bootstrap
Randomization inference	Assignments of states	Sensitivity	30	—	—	−10.8 to −5.4	permutation

Note: Coefficient −8.0 points in every row. Bootstrap B = 9999; randomization inference R = 9999; both seeded.Source: IPUMS USA, ACS 1-year samples 2011–2019 (analysis/usa/acs/part_2010_2014.parquet, part_2015_2019.parquet); expansion dates in _data/expansion_dates.csv (KFF; medicaid.gov); replicate weights REPWTP1–80 (analysis/usa/acs_repwt)

Records. HC1 treats each record as an independent draw, a repetition that no mechanism in this design produces. The design's records come from 964,784 households, about 1.5 per household, so HC1 lands near the household-clustered and replicate values by numerical accident, not because it knows the ACS design, whose strata and weighting it ignores. The interval runs 0.4 percentage points on either side of the estimate.

Households and replicates. Clustering on the ACS household cluster gives a standard error of 0.26 percentage points and the replicate weights give 0.19, against 0.22 for HC1. All three describe survey sampling alone. The replicate value is the smallest, consistent with the stratification and calibration that the replicates reproduce and the other two rules ignore. For the descriptive ledger the replicate interval, −8.0 (95% CI −8.4 to −7.6), is the answer; the household-clustered interval, −8.0 (95% CI −8.5 to −7.5), is a close approximation.

State-years. Clustering by state-year allows shocks shared within a state in a given year but treats a state's 2013 and its 2016 as unrelated. No mechanism works at that level: expansion was assigned to a state for all later years at once, and a state's labor and insurance markets carry over from one year to the next (Bertrand, Duflo, and Mullainathan 2004). The standard error rises to 2.9× the HC1 value and still falls well short of the state-level rules, which answers the prediction: wider than the record-level interval, but closer to it than to the state-clustered one.

States. Clustering by state follows the assignment: −8.0 (95% CI −10.7 to −5.3). The standard error, 1.32 percentage points, is 6.1× the HC1 value and 6.8× the replicate value, and the interval runs 2.7 points on either side.

Few clusters. 30 clusters, 15 of them treated and unequal in size, is not many. Three standard remedies apply (Cameron and Miller 2015; MacKinnon, Nielsen, and Webb 2023). The first, a $t$ reference distribution with $G-1$ degrees of freedom, is already in the CRV1 interval. The second is the CRV3 jackknife, which re-estimates the coefficient leaving out one state at a time: −8.0 (95% CI −11.0 to −5.0). The third is the wild cluster restricted bootstrap, which imposes the null hypothesis and flips the signs of whole states' residuals at random (Cameron, Gelbach, and Miller 2008). With 9,999 replications, inverting it gives −8.0 (95% CI −11.2 to −5.2), with a p-value for no effect of < 0.0001. The three agree closely, which a few large treated states could have prevented.

Randomization inference. A different route asks how unusual the coefficient would be if expansion had gone to 15 of these states at random. Reassigning the cohort label 9,999 times gives a p-value of 0.0001, the smallest that many reassignments can produce, and inverting the test for a constant effect gives −8.0 (95% CI −10.8 to −5.4) (MacKinnon and Webb 2020). Expansion was a political decision, not a lottery, so this is a sensitivity check on the state-clustered interval under a pure assignment model; it does not make the assignment random.

The level came first. The state was chosen from the design before any standard error was computed, and the estimators at that level were then compared for their accuracy with few clusters. The CRV1 interval is not the widest on the ladder (Figure 6.6); a rung with no mechanism behind it can be wide for the wrong reason.

In code: the ladder in pyfixest

The first four rungs are one fit and a change of the vcov argument.

python

import pyfixest as pf
 
# acs: person records for the design states, 2011-2019; D = cohort x (YEAR >= 2014)
fit = pf.feols("uninsured ~ D | STATEFIP + YEAR", data=acs,
               weights="PERWT", vcov="hetero")    # HC1: records independent
fit.vcov({"CRV1": "CLUSTER"})                     # sampling design: ACS households
fit.vcov({"CRV1": "STATEFIP"})                    # assignment design: states
fit.tidy()

Two software details. The pyfixest CRV3 option scales the jackknife differently from the MacKinnon, Nielsen, and Webb definition used here, so it reports a somewhat larger standard error. Its wild bootstrap refuses weighted regressions, so the bootstrap in this chapter runs on the state-year regression multiplied through by the square root of the cell weights, which has the same cluster scores. The replicate-weight rung reruns the fit on each replicate weight; no vcov option produces it.

Laws that never happened

Predict first

Take the 15 comparison states, none of which expanded Medicaid, label 7 of them at random as expanders from 2014, and run the chapter's regression. The true effect is zero. Out of a hundred such assignments, how often will a 5 percent test reject no effect under HC1? Under state clustering?

Bertrand, Duflo, and Mullainathan (2004) settled the argument about serial correlation in difference-in-differences with a device that needs no simulation: assign laws that never happened to randomly chosen states and count how often each standard error rejects the truth. The chapter's cells allow the same check. In the first design, 7 of the 15 comparison states are labelled placebo expanders from 2014. No expansion happened in any of them, so the null holds, and since there are only 6,435 ways to choose the seven, every assignment is run. In the second, 15 of the 30 design states are labelled at random, 4,000 times; the real expansion stays in the outcome and, split between the placebo groups, acts as a state-level shock the placebo regression does not model. Each regression is the ladder's, and each rung is tested against its own reference distribution.

Placebo coefficients across every assignment of seven expanders among the fifteen comparison states

Each bar counts assignments; the markers are the half-widths of a typical 95% interval under two rules

Show data

Placebo coefficient (percentage points)	Assignments
−3.5–−3.0	1
−3.0–−2.5	124
−2.5–−2.0	700
−2.0–−1.5	833
−1.5–−1.0	546
−1.0–−0.5	558
−0.5–0.0	274
0.0–0.5	395
0.5–1.0	665
1.0–1.5	752
1.5–2.0	960
2.0–2.5	575
2.5–3.0	52
Marker: − HC1 half-width	−0.6
Marker: + HC1 half-width	0.6
Marker: − state-clustered half-width	−2.5
Marker: + state-clustered half-width	2.5

Figure 6.7.Across every placebo assignment the coefficient scatters by several percentage points; the record-level interval covers a fraction of that spread and the state-clustered interval most of it.Note: All 6,435 assignments; no expansion happened in these states. Half-widths use the median SE across assignments: t(N − K) for HC1 and t(14) for state clustering.Source: IPUMS USA, ACS 1-year samples 2011–2019 (analysis/usa/acs/part_2010_2014.parquet, part_2015_2019.parquet); expansion dates in _data/expansion_dates.csv (KFF; medicaid.gov)

Laws that never happened: how often each variance rule rejects a true null

Placebo expansion assigned to states at random from 2014; the same regression and rungs as the ladder

Table of rejection rates of a nominal 5 percent test under five variance rules in two placebo designs; the record, household and state-year rules reject a true null far more than 5 percent of the time, and the state-level rules reject near 5 percent.
Placebo design	Variance rule	Independent units	Rejects at 5%	Median SE (points)	Median SE / HC1
15 never-expanded states, 7 placebo expanders (all 6,435 assignments)	HC1 (records independent)	Every record	86.7%	0.28	1.00×
15 never-expanded states, 7 placebo expanders (all 6,435 assignments)	CRV1 by ACS household	Households	82.8%	0.33	1.18×
15 never-expanded states, 7 placebo expanders (all 6,435 assignments)	CRV1 by state-year	State-years	64.0%	0.59	2.04×
15 never-expanded states, 7 placebo expanders (all 6,435 assignments)	CRV1 by state, t(G − 1)	States	12.1%	1.16	3.99×
15 never-expanded states, 7 placebo expanders (all 6,435 assignments)	CRV3 jackknife by state, t(G − 1)	States	0.3%	1.59	5.54×
30 design states, 15 placebo expanders (4,000 seeded draws)	HC1 (records independent)	Every record	86.0%	0.22	1.00×
30 design states, 15 placebo expanders (4,000 seeded draws)	CRV1 by ACS household	Households	83.5%	0.26	1.19×
30 design states, 15 placebo expanders (4,000 seeded draws)	CRV1 by state-year	State-years	48.0%	0.84	3.81×
30 design states, 15 placebo expanders (4,000 seeded draws)	CRV1 by state, t(G − 1)	States	8.3%	1.94	8.87×
30 design states, 15 placebo expanders (4,000 seeded draws)	CRV3 jackknife by state, t(G − 1)	States	5.0%	2.24	10.17×

Note: In the first design no expansion happened in any state, so the null is true by construction. In the second the real expansion stays in the outcome and acts as a state-level shock the placebo regression does not model. Tests use t(N − K) for HC1 and t(G − 1) for the clustered rules, as pyfixest does.Source: IPUMS USA, ACS 1-year samples 2011–2019 (analysis/usa/acs/part_2010_2014.parquet, part_2015_2019.parquet); expansion dates in _data/expansion_dates.csv (KFF; medicaid.gov)

In the first design the placebo coefficients have a standard deviation of 1.57 percentage points. The median HC1 standard error is 0.28, and HC1 rejects the true null in 86.7% of assignments; clustering by household rejects in 82.8% and by state-year in 64.0%. These rules describe a repetition in which only the sample changes, and between two samples a state's uninsured share barely moves. The placebo repetition changes which states carry the label, and the states' own trajectories differ by far more than survey noise (Figure 6.5). Clustering by state, with a median standard error of 1.16 points and a $t$ distribution on 14 degrees of freedom, rejects in 12.1%: nearer 5 percent than any rung below it, and still above it, because 15 clusters with seven treated is the few-cluster case the ladder warned about. The CRV3 jackknife, median 1.59 points, rejects in 0.3%, erring the other way. With the 30 design states the same rungs give 86.0% for HC1, 8.3% for state clustering, and 5.0% for the jackknife: twice the clusters brings the state-level rules to or near the nominal rate and leaves the record-level rules where they were.

The check does not say what the standard error of the real coefficient is. It says which rules describe a repetition like the one the policy question asks about, and it shows the few-cluster corrections doing the work the ladder assigned them. Chapter 7 repeats the exercise on simulated data, where the mechanism is fixed by construction.

Filling in the worksheet

Predict first

Before opening the worksheet, decide at which level you would cluster this chapter's policy comparison and which standard error you would report. Then enter the design and see whether the worksheet agrees, and which parts of your answer it says rest on assumptions.

Where does the common uncertainty come from?

Describe the target, how units were sampled, and how treatment was assigned. The worksheet derives a clustering level from the design, with its assumptions and alternatives.

Start from an example:

Recommended clustering level

Cluster at the level where treatment is assigned.Why: Every unit in a cluster shares one assignment, so there are 51 independent assignment units, not one per record.

Variance estimator

Cluster-robust (CR1, Liang–Zeger) variance with t(G − 1) critical values.Why: The design has enough assignment clusters for the cluster-robust approximation.

What this rests on

The descriptions of sampling and assignment above are accurate.Why: The clustering level follows from how the data came to be, not from which standard error is largest.
Clusters do not affect one another (no spillovers across clusters).Why: Cluster-level inference treats clusters as independent units.
Sampling units (for example survey PSUs) sit inside the assignment clusters.Why: In a nested hierarchy, clustering at the coarser, assignment level also covers the sampling clustering.
Treatment changes within clusters (for example over time in a panel).Why: Cluster fixed effects absorb anything constant within a cluster, including a treatment that never switches on; they do not remove the need to cluster at the assignment level.
Many clusters, not many records, drive the large-sample approximation.Why: Cluster-robust inference is asymptotic in the number of clusters.

Also report (sensitivity)

Heteroskedasticity-robust (EHW) SEs, shown only to measure how much clustering matters.Why: They ignore the clustered assignment and understate the uncertainty; they are not the result.

Few-cluster check

51 assignment clusters: CR1 with t(G − 1) critical values is usually adequate. Still compare with the wild cluster bootstrap.Why: The effective number of clusters is smaller than G when cluster sizes are very unequal or only a few clusters are treated.

Notes

All clusters are observed and they are the target, so the uncertainty is about assignment, not sampling.Why: The cluster-robust variance is then conservative when effects vary across clusters, as the Neyman variance is in an experiment.
This worksheet addresses inference conditional on the design.Why: It does not establish that the coefficient is causal; identification is a separate argument (Chapter 6).

Rules follow Abadie, Athey, Imbens, and Wooldridge (2023): cluster when sampling or assignment is clustered, at the level of that mechanism; in a nested hierarchy the coarser relevant level typically governs. The output is a starting point for an argument, not a verdict.

What is held fixed. The estimand (an average treatment effect from a regression) and the data. Only the description of the design changes, and with it the source of uncertainty the standard error must reflect.

Start from the preset State policy, survey records and set the number of clusters to this chapter's design:

Inferential target. The observed units: every state that meets the cohort rules is in the data, and the question is about those states.
Sampling mechanism. Individuals sampled within every cluster; the ACS samples addresses in every state.
Assignment mechanism. Whole clusters: everyone in a state shares its decision.
Nesting. Yes; addresses and households lie inside states.
Cluster fixed effects. Yes, state fixed effects alongside year effects.
Number of clusters. 30 states, 15 of them treated.

The worksheet's answer is to cluster at the state, the level where treatment is assigned, and, with fewer than about 50 assignment clusters, to use the wild cluster restricted bootstrap or CRV3 with $t(G-1)$ critical values rather than CRV1 with normal ones. It lists the assumptions that answer rests on: states do not affect one another; households nest inside states, so the state level also covers the survey's clustering; and treatment switches on within states over time, since state fixed effects absorb anything that never changes within a state. Because every state is observed and the states are the target, it adds that the uncertainty is about assignment rather than sampling, so the cluster-robust variance is conservative if effects differ across states.

Two inputs carry the argument. Switch the assignment mechanism to the individual level and the recommendation becomes no clustering adjustment at all: when treatment varies from person to person, correlated outcomes within states do not by themselves call for clustering (Abadie et al. 2023). Lower the number of clusters below 10 and it moves to randomization inference. The cutoffs, 50 and 10, are rules of thumb (MacKinnon, Nielsen, and Webb 2023; Cameron and Miller 2015), prompts to compare estimators rather than bright lines. The worksheet does not decide whether the coefficient is causal; no choice of standard error settles that.

Two questions, two intervals

The opening set a record-level interval beside a state-clustered one, but those are not the two answers; the descriptive question and the policy question each have their own. If the question is how much more uninsurance fell in these 15 states than in these 15, with their policies taken as given, only the Census Bureau's sample would change in a repetition, and the replicate interval, −8.0 (95% CI −8.4 to −7.6), describes it. If the question is what expansion did, a repetition would also redraw which states expanded, and for that question there are 30 units of information, not 1,484,005; the state-clustered interval describes it, with the few-cluster checks beside it. Neither interval is a cautious version of the other.

Weighting each state-year cell equally instead of by population targets the average state rather than the average low-income adult, and gives −8.4. That is a change of estimand, the subject of Chapter 5, not a change of variance.

The principle does not depend on states or on the ACS: a standard error describes a repetition, and clustering asks which observations would move together in it, through the sampling of units or the assignment of the regressor. Chapter 7 carries it across a classroom lottery, a state variable in an individual regression, shocks that travel over the map or hit every unit at once, and a shift-share exposure.

Note

A new case

A study observes every state, not a sample of them, and a policy was adopted by some states and not others. With no sampling of states, is there any uncertainty left for a standard error to describe?

One answer

Yes. The survey still samples households within each state, so each state's outcome carries sampling error, though here it is small. The larger uncertainty is about assignment: which states adopted could have been otherwise, and the policy question treats that assignment as the repetition. Observing every state removes the sampling argument for clustering by state, not the assignment argument (Abadie et al. 2023). Uncertainty about the design itself, the cohort rules and parallel trends, is in no standard error at all.

What we learned about the question

In the 15 states that expanded Medicaid on January 1, 2014, and had not covered low-income adults earlier, the uninsured share among adults 19 to 64 at or below 138 percent of poverty changed by −8.0 (95% CI −10.7 to −5.3) percentage points relative to 15 states that had not expanded by 2019, comparing 2014–2019 with 2011–2013. The ACS measures each state's change precisely. States differ widely in how much their uninsurance fell, and that variation, not survey noise, limits the comparison: expansions assigned at random to states that never expanded are declared significant by HC1 86.7% of the time and by state clustering 12.1%.

What the method changed

Not the coefficient. The method changed the number of independent units, from 1,484,005 records to 30 states, and made the policy standard error 6.8× the replicate standard error that answers the descriptive question. It also separated two targets that share one number: a descriptive contrast, whose uncertainty is survey sampling, and a policy effect, whose uncertainty includes the assignment of expansion across states.

What this analysis does not establish

That expansion caused the difference. Parallel trends, no anticipation, and no spillovers are assumed, not tested. Extending the pre-period back to 2008 gives a 2008 coefficient of 2.3 (95% CI −0.3 to 4.9); the 2011 start sets those recession years aside without explaining them. Idaho and Nebraska voted to expand in 2018 and began coverage in 2020, so anticipation in 2019 is possible in the comparison group.
That the number describes Medicaid expansion in general. It covers one cohort in one window, and a stricter cohort rule moves it by more than its state-clustered standard error: the interval describes uncertainty given the design, not uncertainty about the design. Later adopters and years after 2019 are outside it.
That the population is fixed. Eligibility here is defined by measured family income relative to the Census poverty threshold, a proxy for Medicaid's own test. Expansion could change income, work, or household composition, and people move between states.
That states are independent of one another. State clustering allows any correlation within a state and none between states. Shocks shared by neighboring states would make even the state-clustered interval too narrow.
That assignment was random. Randomization inference describes what random assignment would imply. States chose.
That the state-clustered interval has 95 percent coverage. With 15 clusters the placebo check found CRV1 by state rejecting a true null 12.1% of the time and the jackknife 0.3%. The two bracket the nominal rate; the wild cluster bootstrap interval is the one to report.