RepoRoulette: Randomly Sample GitHub Repositories

RepoRoulette: Randomly Sample GitHub Repositories.

A Python library for randomly sampling GitHub repositories using multiple methods: - ID-based sampling: Probes random repository IDs - Temporal sampling: Weighted sampling based on repository activity by time period - BigQuery sampling: Advanced querying using Google BigQuery’s GitHub dataset - GitHub Archive sampling: Event-based sampling from GitHub Archive files

Example

>>> from reporoulette import sample
>>> results = sample(method='temporal', n_samples=10)
>>> print(f"Found {len(results['samples'])} repositories")

class reporoulette.BigQuerySampler(credentials_path: str | None = None, project_id: str | None = None, seed: int | None = None, log_level: int = 20)

Bases: BaseSampler

Sample repositories using Google BigQuery’s GitHub dataset.

This sampler leverages the public GitHub dataset in Google BigQuery to efficiently sample repositories with complex criteria and at scale. The population is repositories generating GH Archive events on the sampled days, so the sample is biased toward active repositories.

get_languages(repos: list[dict[str, Any]]) → dict[str, list[dict[str, Any]]]

Retrieve language information for a list of repositories.

sample(n_samples: int = 100, population: str = 'all', **kwargs: Any) → list[dict[str, Any]]

Sample repositories using BigQuery.

Parameters:

• n_samples – Number of repositories to sample

• population – Type of repository population to sample from (‘all’ or ‘active’)

• **kwargs – Additional filtering criteria

Returns:

List of repository dictionaries

sample_active(n_samples: int = 100, created_after: str | datetime | None = None, created_before: str | datetime | None = None, languages: list[str] | None = None, **kwargs: Any) → list[dict[str, Any]]

Sample repositories with recent commit activity.

Parameters:

• n_samples – Number of repositories to sample

• created_after – Filter commits after this timestamp

• created_before – Filter commits before this timestamp

• languages – List of programming languages to filter by (uses github_repos.languages)

• **kwargs – Additional filter criteria

Returns:

List of repository dictionaries

sample_by_day(n_samples: int = 100, days_to_sample: int = 10, repos_per_day: int = 50, years_back: int = 10, **kwargs: Any) → list[dict[str, Any]]

Sample repositories using a day-based approach with GitHub Archive tables.

class reporoulette.GHArchiveSampler(token: str | None = None, seed: int | None = None, log_level: int = 20)

Bases: BaseSampler

Sample repositories by downloading and processing GH Archive files.

This sampler randomly selects days from GitHub’s event history, downloads the corresponding archive files, and extracts repository information. With the default CreateEvent filter the population is repositories created on the sampled days; with other event types it is an activity-biased event population.

gh_sampler(n_samples: int = 100, days_to_sample: int = 5, repos_per_day: int = 20, years_back: int = 10, event_types: list[str] | None = None, hours_per_day: int | None = None, **kwargs: Any) → list[dict[str, Any]]

Sample repositories from GH Archive’s hourly files.

The population sampled depends on hours_per_day. With the default (None), all 24 hourly files of each sampled day are processed, so each day’s population is exactly the repositories with matching events that day. With hours_per_day=H, only H randomly chosen hours are downloaded (H/24 of the bandwidth), and the population becomes repositories with matching events in the sampled hours - repos active in low-traffic hours are then over-represented relative to the full-day population, the same bias structure the per-day cap already introduces across days.

Parameters:

• n_samples – Target number of repositories to sample

• days_to_sample – Number of random days to sample

• repos_per_day – Maximum repositories to sample per day

• years_back – How many years to look back

• event_types – Types of GitHub events to consider. Note: GitHub’s Events API payload change of 2025-10-07 removed repository CreateEvents from the public feed, so with the default event_types no repository created after that date can be sampled; days after it have an empty population.

• hours_per_day – Number of random hours (of 24) to download per day; None processes the full day

• **kwargs – Additional filters to apply

Returns:

List of repository data

sample(n_samples: int = 100, **kwargs: Any) → list[dict[str, Any]]

Sample repositories using the GH Archive approach.

This is the implementation of the abstract method from BaseSampler, which delegates to the gh_sampler method with the provided parameters.

Parameters:

• n_samples – Number of repositories to sample

• **kwargs – Additional parameters to pass to gh_sampler

Returns:

List of repository data

class reporoulette.IDSampler(token: str | None = None, min_id: int = 1, max_id: int = 1300000000, rate_limit_safety: int = 100, seed: int | None = None, log_level: int = 20)

Bases: BaseSampler

Sample repositories using random ID probing.

This sampler generates random repository IDs within a specified range and attempts to retrieve repositories with those IDs from GitHub. The resulting sample is approximately uniform over all existing public repositories with ID <= max_id.

DEFAULT_MAX_ID = 1300000000

api_base_url: str

attempts: int

default_timeout: int

logger: Logger

max_id: int

max_retries_on_rate_limit: int

min_id: int

min_request_interval: float

rate_limit_resource: str

rate_limit_safety: int

results: list[dict[str, Any]]

sample(n_samples: int = 10, min_wait: float = 0.1, max_attempts: int = 1000, **kwargs: Any) → list[dict[str, Any]]

Sample repositories by trying random IDs.

Parameters:

• n_samples – Number of valid repositories to collect

• min_wait – Minimum wait time between API requests

• max_attempts – Maximum number of IDs to try

• **kwargs – Additional filters to apply (filtering happens during collection)

Returns:

List of repository data

success_count: int

token: str | None

update_max_id() → int

Refresh max_id from the newest repository visible on GitHub.

Makes one search API call for repositories created in the last day and sets max_id to the highest ID observed (approximately the current ID ceiling). Falls back to the existing max_id with a warning if the call fails.

Returns:

The (possibly updated) max_id

class reporoulette.TemporalSampler(token: str | None = None, start_date: datetime | str | None = None, end_date: datetime | str | None = None, rate_limit_safety: int = 5, seed: int | None = None, years_back: int = 10, log_level: int = 20)

Bases: BaseSampler

Sample repositories by randomly selecting days and fetching repos updated in those periods.

This sampler selects random days within a specified date range, weights them by repository count, and retrieves repositories with proportional sampling. The population is repositories pushed on the sampled days, so the sample is biased toward actively maintained repositories, and the Search API’s 1,000-results-per-query cap limits coverage on high-activity days.

sample(n_samples: int = 100, days_to_sample: int = 10, per_page: int = 100, min_wait: float = 1.0, min_stars: int = 0, min_size_kb: int = 0, language: str | None = None, max_attempts: int = 100, **kwargs: Any) → list[dict[str, Any]]

Sample repositories by randomly selecting days with weighting based on repo count.

Parameters:

• n_samples – Target number of repositories to collect. Collection proceeds a full search page at a time, so the returned list can exceed this target (it is a lower bound, not an exact size).

• days_to_sample – Number of random days to initially sample for count assessment

• per_page – Number of results per page (max 100)

• min_wait – Minimum wait time between API requests

• min_stars – Minimum number of stars (0 for no filtering)

• min_size_kb – Minimum repository size in KB (0 for no filtering)

• language – Programming language to filter by

• max_attempts – Maximum collection-loop iterations before giving up

• **kwargs – Additional filters to apply

Returns:

List of repository data

reporoulette.sample(method: str = 'temporal', n_samples: int = 50, token: str | None = None, **kwargs: Any) → dict[str, Any]

Sample repositories using the specified method.

Parameters:

• method – Sampling method (‘id’, ‘temporal’, ‘archive’, or ‘bigquery’)

• n_samples – Number of repositories to sample

• token – GitHub Personal Access Token (not used for BigQuery)

• **kwargs – Additional parameters specific to each sampler

Returns:

Dictionary with sampling results and stats

Raises:

ValueError – If an unknown sampling method is provided

RepoRoulette provides multiple methods for randomly sampling GitHub repositories:

• ID-based sampling: Probes random repository IDs

• Temporal sampling: Weighted sampling based on repository activity by time period

• BigQuery sampling: Advanced querying using Google BigQuery’s GitHub dataset

• GitHub Archive sampling: Event-based sampling from GitHub Archive files

Quick Start

from reporoulette import sample

# Sample 10 repositories using temporal sampling
results = sample(method='temporal', n_samples=10)
print(f"Found {len(results['samples'])} repositories")

Contents:

• Sample Function
  • sample()
  • Usage Examples
  • Return Format
• ID Sampler
  • IDSampler
  • Advantages
  • Disadvantages
  • Usage Example
• Temporal Sampler
  • TemporalSampler
  • Advantages
  • Disadvantages
  • Usage Example
• BigQuery Sampler
  • BigQuerySampler
  • Advantages
  • Disadvantages
  • Usage Example
• GitHub Archive Sampler
  • GHArchiveSampler
  • Advantages
  • Disadvantages
  • Usage Example

Indices and tables

• Index

• Module Index

• Search Page