Randomization Validation Report

Generated 2026-07-24 15:23 by scripts/validate_randomization.py. All tests at alpha = 0.05. Chi-square critical values use the Wilson-Hilferty approximation; KS uses the asymptotic two-sample critical value. Aggregating draws across seeded runs treats draws as independent, which is a good approximation for sample sizes far below the population size.

P1 - GH Archive ground-truth inclusion test (GHArchiveSampler)

Population: all 7,294 repositories created in hour 12 UTC of 2023-06-15 (fully enumerated from the public archive). 100 seeded replays of 100 draws each (10,000 total draws) through the real sampler code path.

Test	Statistic	Critical (5%)	Result
Equal inclusion probability (100 hash buckets, df=99)	105.3	123.2	PASS
Creation minute-of-hour marginal vs population (df=59)	51.8	77.9	PASS
Owner-initial marginal vs population	27.6	38.9	PASS

P3 - Seed stability (pairwise KS across seeds)

4950 pairwise two-sample KS tests on within-day creation times across independent seeds: 250 rejections (5.1%; expectation under uniform sampling is about 5%). PASS

P2 - Live IDSampler audit

2,000 live probes against the real GitHub API, 747 hits (37.4% success rate).

Test	Statistic	Critical (5%)	Result
Probed IDs uniform over 10 deciles (df=9)	9.24	16.90	PASS
created_at monotone in ID (inversion rate)	0.00%	< 1%	PASS
Live probe-sequence reproducibility (same seed, 2 runs)	identical	identical	PASS

Hit rate by ID decile (existing-repo density over GitHub’s history): 42.1%, 50.0%, 37.9%, 38.1%, 31.6%, 34.9%, 35.0%, 39.3%, 34.3%, 29.7%.

Coverage gap: newest observed repo ID is 1,310,925,406 vs the default max_id 850,000,000 - the default excludes 35.2% of the current ID space (newest repositories). Pass max_id explicitly for full coverage.

P4 - Temporal sampler bias, quantified (measurement, not pass/fail)

Temporal sample n=200 vs near-uniform ID-sample benchmark n=747:

Metric	TemporalSampler	IDSampler benchmark
Mean stars	0.4	0.2
Median stars	0	0
% with >= 1 star	12%	9%
% pushed in last 90 days	0%	0%
Top languages	JavaScript (36), HTML (28), Python (20), TypeScript (16), Jupyter Notebook (8)	JavaScript (105), HTML (77), Java (53), Python (49), Jupyter Notebook (23)

Search-cap coverage: mean fraction of each sampled day’s repositories reachable under the 1,000-result cap: 2.7% (per-day counts: 2017-02-25: 19,301, 2017-07-18: 32,509, 2017-10-10: 34,576, 2019-07-22: 33,353, 2019-07-27: 21,277, 2021-02-18: 66,816, 2025-03-10: 157,895, 2025-12-20: 151,606).

P5 - BigQuery live check (BigQuerySampler)

Check	Result
Returns real data (was always empty before the invalid-SQL fixes)	49 repos - PASS
Same seed, two live runs identical	PASS
Duplicate repositories in output	0 - PASS
Days used / allocation vs day size correlation	5 days, r = 1.00

P6 - Live API contract checks

Check	Result
GH Archive publishes hourly files only (daily URL 404s, hourly 200; probed 2026-07-22)	daily=404, hourly=200 - PASS
Event schema fields used by GHArchiveSampler present	first 500 events well-formed - PASS
Repository CreateEvents ended with GitHub’s 2025-10-07 Events API change (present before, absent after)	1878 in 50k events of 2025-08-15 vs 0 recently - PASS
/repositories/{id} returns the fields IDSampler maps	HTTP 200 - PASS
Temporal population claim: pushed_at inside the sampled day	100 repos, 100% in window - PASS

P7 - BigQuery dry-run validation (free)

Every query shape the sampler can emit, compiled by BigQuery without execution (dry runs cost nothing and would have caught all three SQL-validity bugs in this audit). Estimated scan per query:

Query	Valid	Est. scan
count (5 days, pruned wildcard)	PASS	0.52 GB
day sample	PASS	0.18 GB
combined (2 days, UNION ALL)	PASS	0.18 GB
active (no languages)	PASS	99.45 GB
active (languages)	PASS	99.59 GB
get_languages	PASS	0.20 GB