API Reference

Main Functions

The lost_years package provides three main functions for calculating expected years of life lost:

lost_years_ssa

lost_years.lost_years_ssa(df: DataFrame, cols: dict[str, str] | None = None, age_tolerance: float | None = 1.0, year_tolerance: float | None = None) → DataFrame

Append SSA life expectancy to the input DataFrame.

Matches each row on age, sex and year using the column names given by cols. A row whose age or year lies further from the packaged table than the tolerances allow gets a missing life expectancy and a ssa_match_status saying so, rather than the nearest available figure passed off as the answer.

Parameters:

• df – Pandas DataFrame containing the input data.

• cols – Column mapping for age, sex, and year in DataFrame. If None, uses the default mapping {'age': 'age', 'sex': 'sex', 'year': 'year'}.

• age_tolerance – How far, in years of age, the match may sit from the requested age. None accepts any distance.

• year_tolerance – How far, in calendar years, the match may sit from the requested year. None accepts any distance.

Returns:

‘ssa_age’, ‘ssa_year’, ‘ssa_life_expectancy’, ‘ssa_match_status’

Return type:

Pandas DataFrame with life expectancy columns

lost_years_hld

lost_years.lost_years_hld(df: DataFrame, cols: dict[str, str] | None = None, subpopulations: bool = False, max_ex_discrepancy: float | None = 2.0, year_tolerance: float | None = None) → DataFrame

Append HLD life expectancy to the input DataFrame.

Every input row gets exactly one output row by default, taken from the whole-country total-population life table chosen by select_life_expectancy(). hld_match_status says why a lookup returned nothing, and hld_n_candidates says how many equally eligible life tables the tie-break had to choose between.

Parameters:

• df – Pandas DataFrame containing the input data.

• cols – Column mapping for country, age, sex, and year in DataFrame. None for default mapping: {‘country’: ‘country’, ‘age’: ‘age’, ‘sex’: ‘sex’, ‘year’: ‘year’}.

• subpopulations – Emit one row per available sub-population (region, urban/rural, ethnicity, socio-demographic group) instead of the single national total. The output then has more rows than the input.

• max_ex_discrepancy – Quarantine threshold in years for the gap between HLD’s recalculated and published life expectancy. None serves the quarantined life tables anyway.

• year_tolerance – How many years outside a life table’s period a query may fall before the table stops being an acceptable answer. None, the default, requires the period to contain the requested year; setting it records the distance in hld_match_status.

Returns:

The input DataFrame with the HLD columns appended.

Note

Propagates lost_years.TableUnavailableError when no HLD table has been downloaded yet. HLD is not shipped in the wheel; run lost_years update --source hld once to install it.

lost_years_who

lost_years.lost_years_who(df: DataFrame, cols: dict[str, str] | None = None, year_tolerance: float | None = None) → DataFrame

Append WHO life expectancy at birth to the input DataFrame.

Matches each row on country, sex and year using the column names given by cols. There is deliberately no age dimension: the packaged WHO indicator is life expectancy at birth, so the returned column is named for what it holds and an age mapping is refused.

Parameters:

• df – Pandas DataFrame containing the input data.

• cols – Column mapping for country, sex, and year in DataFrame. None for default mapping: {‘country’: ‘country’, ‘sex’: ‘sex’, ‘year’: ‘year’}.

• year_tolerance – How far, in calendar years, the match may sit from the requested year. None accepts any distance.

Returns:

‘who_country’, ‘who_sex’, ‘who_year’, ‘who_life_expectancy_at_birth’, ‘who_match_status’.

Return type:

Pandas DataFrame with WHO data columns

Raises:

ValueError – If cols maps an age column. The WHO table cannot answer an age-specific question.

Note

Propagates lost_years.TableUnavailableError when no WHO table has been downloaded yet; run lost_years update --source who once to install it.

Module Details

SSA Module

SSA (US Social Security Administration) period life tables for lost_years.

class lost_years.ssa.LostYearsSSAData

SSA life-table lookup, caching the packaged table on first use.

lost_years.ssa.main(argv: list[str] = ['-W', '-b', 'html', 'docs/source', '_site']) → int

Run the lost_years_ssa command line interface.

Parameters:

argv – Command line arguments, defaulting to the process arguments.

Returns:

0 on success, -1 when a required column is missing.

HLD Module

HLD (Human Life-Table Database) module for lost_years package.

The pooled HLD file is a collection of life tables as they were published, not one estimate per country-year: a single country-year is often covered by several tables that differ in geography (whole country vs. a province), sub-population (urban/rural, ethnicity, education), source publication, table type and reference period. Choosing a row therefore needs an explicit, documented rule; select_life_expectancy() is that rule.

lost_years.hld.read_hld() → DataFrame

Read the derived HLD table.

The file is built by lost_years update --source hld, which is where the upstream repairs live: sub-population codes read as text, HLD’s literal NA region mapped to the whole-country code, and each life table’s own largest |e(x) - e(x)Orig| precomputed into ex_discrepancy. Reading is therefore a plain load with no cleaning, and the manifest beside the file says which upstream release it came from.

Returns:

Every HLD row eligible to be looked up, one per age interval of one published life table.

lost_years.hld.eligible_tables(table: DataFrame, national_only: bool = True, max_ex_discrepancy: float | None = 2.0) → DataFrame

Keep only the HLD life tables that are allowed to answer a lookup.

Three filters run before any query is answered:

1. Region == Residence == Ethnicity == SocDem == '0' keeps the whole-country total population. Skipped when national_only is False.

2. TypeLT == 2 is dropped. Type 2 is HLD’s own abridgement of the type 1 complete table from the same source, so it never carries information the finer table does not already have.

3. Tables whose recalculated and published life expectancies disagree by more than max_ex_discrepancy years are quarantined.

Parameters:

• table – Every HLD row, as returned by read_hld().

• national_only – Keep only whole-country, total-population life tables.

• max_ex_discrepancy – Quarantine threshold in years. None serves the quarantined tables anyway.

Returns:

The eligible subset.

lost_years.hld.load_hld_table(national_only: bool = True, max_ex_discrepancy: float | None = 2.0) → DataFrame

Load the HLD life tables that are eligible to answer a lookup.

The result is cached, so repeated calls with the same arguments neither re-read nor re-filter the 2.2M-row table.

Parameters:

• national_only – Keep only whole-country, total-population life tables.

• max_ex_discrepancy – Quarantine threshold in years. None serves the quarantined tables anyway.

Returns:

The eligible subset of the HLD table.

lost_years.hld.empty_result(status: str) → dict[str, Any]

Build an all-missing output record.

Parameters:

status – Why no life expectancy could be returned.

Returns:

Mapping of output column name to value.

lost_years.hld.select_life_expectancy(table: DataFrame, country: str, year: float, sex: str, age: float, year_tolerance: float | None = None) → dict[str, Any]

Pick exactly one HLD row for one (country, year, sex, age) query.

The rule, in order:

1. Country: exact ISO-3166-1 alpha-3 match, case-insensitive. No substring or regular-expression matching – US is not a country code and must not match AUS.

2. Period containment: keep tables with Year1 <= year <= Year2. 2,482 country-years in HLD are reachable only through a multi-year period table, so containment is required, not optional. A year no table covers returns nothing unless year_tolerance is set, in which case the nearest period within the tolerance is used and hld_match_status records the distance.

3. Narrowest period: of those, keep the tables with the smallest Year2 - Year1, so a 1980 table beats a 1976-1980 table for 1980.

4. Age interval: keep the row whose interval [Age, Age + AgeInt) contains the requested age. AgeInt == 99 marks the open top interval; the 37 upstream rows with a negative AgeInt cannot define an interval and are dropped.

5. Tie-break convention: highest Version, then highest Ref-ID, then latest Year1, then lowest TypeLT. Version is HLD’s own revision counter, so the highest is the most revised table; Ref-ID rises as sources are added, so the highest is the most recently added source. The last two keys exist only to make the order total. About 17% of country-year cells still reach this step with more than one candidate, which is why the count is reported in hld_n_candidates rather than hidden.

Parameters:

• table – Eligible HLD rows, as returned by load_hld_table().

• country – ISO-3166-1 alpha-3 country code.

• year – Calendar year.

• sex – “M” or “F”.

• age – Exact age in years.

• year_tolerance – How many years outside a table’s period the query may fall before the table stops being an acceptable answer. None, the default, requires containment.

Returns:

Mapping of output column name to value. hld_match_status is “ok” when a row was selected and says what went wrong otherwise.

lost_years.hld.select_subpopulations(table: DataFrame, country: str, year: float, sex: str, age: float, year_tolerance: float | None = None) → list[dict[str, Any]]

Run select_life_expectancy() once per available sub-population.

Each sub-population is resolved on its own, so every returned row is as unambiguous as the national-total row is by default; what varies is how many rows come back.

Parameters:

• table – Eligible HLD rows including sub-populations.

• country – ISO-3166-1 alpha-3 country code.

• year – Calendar year.

• sex – “M” or “F”.

• age – Exact age in years.

• year_tolerance – Passed through to select_life_expectancy().

Returns:

One record per sub-population with a life table for this query, or a single all-missing record when there is none.

lost_years.hld.normalise_sex(value: Any) → str

Map an input sex value onto “M” or “F”.

Parameters:

value – Raw value from the input DataFrame.

Returns:

“M”, “F”, or “” when the value is not a recognised token.

class lost_years.hld.LostYearsHLDData

HLD data handler for life table information.

lost_years.hld.main(argv: list[str] = ['-W', '-b', 'html', 'docs/source', '_site']) → int

Run the lost_years_hld command line interface.

Parameters:

argv – Command line arguments, defaulting to the process arguments.

Returns:

0 on success, -1 when a required column is missing or no HLD table has been installed.

WHO Module

WHO (World Health Organization) life expectancy tables for lost_years.

The WHO table is GHO indicator WHOSIS_000001, life expectancy at birth. It has no age dimension: one value per population, year and sex. The lookup therefore answers questions about age 0 only, and says so in the name of the column it returns. Asking it for remaining life expectancy at a given age is a question it cannot answer, so an explicit age mapping raises instead of being quietly ignored; use lost_years.lost_years_hld() for that.

class lost_years.who.LostYearsWHOData

WHO life-table lookup, caching the packaged table on first use.

lost_years.who.main(argv: list[str] = ['-W', '-b', 'html', 'docs/source', '_site']) → int

Run the lost_years_who command line interface.

Parameters:

argv – Command line arguments, defaulting to the process arguments.

Returns:

0 on success, -1 when a required column is missing or no WHO table has been installed.

Data management

Where life tables live on disk, and how one is replaced without tearing.

Only the SSA table ships inside the wheel. HLD is redistributed under terms that ask users to fetch their own copy, and the WHO table is large enough that shipping it makes the package stale the day it is published, so both are downloaded by lost_years update into a per-user data directory.

Every derived table is a Parquet file with an explicit Arrow schema, and every derived table has a manifest beside it recording where it came from. A table is only ever installed by writing it under a temporary name in its final directory and renaming it into place, so a failed or interrupted update leaves the previous copy exactly as it was.

exception lost_years.datasets.TableUnavailableError

A source has neither a shipped nor a downloaded table.

exception lost_years.datasets.ValidationError

A candidate table failed a check, so it was not installed.

lost_years.datasets.data_dir() → Path

Return the directory holding downloaded life tables.

Returns:

$LOST_YEARS_DATA_DIR when set, otherwise the per-user data directory for this platform.

lost_years.datasets.shipped_path(source: str, filename: str) → Path

Return where filename would sit inside the installed package.

Parameters:

• source – Source name, e.g. "ssa".

• filename – Table file name, e.g. "ssa.parquet".

Returns:

Path inside the import package. It need not exist.

lost_years.datasets.installed_path(source: str, filename: str) → Path

Return where filename would sit in the user’s data directory.

Parameters:

• source – Source name, e.g. "hld".

• filename – Table file name, e.g. "hld.parquet".

Returns:

Path under data_dir(). It need not exist.

lost_years.datasets.resolve(source: str, filename: str) → Path

Find the table to read for one source.

A downloaded table always wins over a shipped one: running lost_years update is how a user replaces a stale packaged table.

Parameters:

• source – Source name.

• filename – Table file name.

Returns:

Path to an existing Parquet file.

Raises:

TableUnavailableError – When neither location holds the table.

lost_years.datasets.sha256(path: Path) → str

Return the lowercase SHA-256 digest of a file.

Parameters:

path – File to digest.

Returns:

64-character hexadecimal digest.

lost_years.datasets.describe_schema(schema: Schema) → list[dict[str, Any]]

Render an Arrow schema as plain JSON-serialisable records.

Parameters:

schema – Schema of a derived table.

Returns:

One record per field, with its name, Arrow type and nullability.

lost_years.datasets.build_manifest(*, source: str, title: str, home_url: str, source_url: str, license_terms: str, upstream_release: str, built_from: str, table: Path, raw_sha256: str | None, notes: dict[str, Any]) → dict[str, Any]

Describe a derived table and where it came from.

Parameters:

• source – Source name.

• title – Human-readable name of the upstream database.

• home_url – Landing page a user should cite and read the terms on.

• source_url – URL the raw artifact was downloaded from.

• license_terms – Redistribution terms of the upstream data.

• upstream_release – Upstream’s own identifier for this release.

• built_from – "download", or the local path the table was built from when the download was bypassed.

• table – The built Parquet file.

• raw_sha256 – Digest of the raw artifact the table was built from, or None when the table was built from a local file of unknown origin.

• notes – Source-specific build facts, such as how many malformed upstream lines were dropped.

Returns:

The manifest, ready to serialise.

lost_years.datasets.manifest_for(table: Path) → Path

Return the manifest path beside a table.

Parameters:

table – Path to a derived table.

Returns:

Sibling path with the manifest suffix.

lost_years.datasets.read_manifest(table: Path) → dict[str, Any] | None

Read the manifest beside a table.

Parameters:

table – Path to a derived table.

Returns:

The manifest, or None when there is none.

lost_years.datasets.install(table: Path, manifest: dict[str, Any], destination: Path) → Path

Move a validated table and its manifest into place atomically.

Both files are written under a temporary name in the destination directory, so the rename that publishes them is a same-filesystem Path.replace: either the old table or the new one is visible to a concurrent reader, never a half-written file. The table is published first, because it is the file readers require; a crash between the two renames leaves a manifest that no longer matches, which lost_years status reports.

Parameters:

• table – Validated Parquet file, anywhere on disk.

• manifest – Manifest to write beside it.

• destination – Directory to install into.

Returns:

The installed table path.

Install a life table: download, build, validate, then swap it into place.

Nothing is ever written over a working table until a candidate has passed every check, so an upstream that truncates a file, renames a column, or ships a table that no longer reproduces the published figures leaves the user with what they already had rather than with quietly wrong numbers.

class lost_years.update.UpdateResult(source: str, path: Path, manifest: dict[str, Any], replaced: bool)

What one lost_years update of a single source did.

source: str

path: Path

manifest: dict[str, Any]

replaced: bool

__init__(source: str, path: Path, manifest: dict[str, Any], replaced: bool) → None

class lost_years.update.SourceStatus(source: str, kind: str, path: Path | None, installed_release: str | None, fetched_at: str | None, rows: int | None, upstream_release: str | None, state: str, note: str)

What is installed for one source, and whether upstream has moved on.

source: str

kind: str

path: Path | None

installed_release: str | None

fetched_at: str | None

rows: int | None

upstream_release: str | None

state: str

note: str

__init__(source: str, kind: str, path: Path | None, installed_release: str | None, fetched_at: str | None, rows: int | None, upstream_release: str | None, state: str, note: str) → None

lost_years.update.update(name: str, *, from_file: Path | None = None, destination: Path | None = None) → UpdateResult

Fetch, build, validate and install one source’s table.

Parameters:

• name – Source name.

• from_file – Build from this local artifact instead of downloading. The escape hatch for hosts that refuse automated clients, and the way to rebuild from an archived copy.

• destination – Directory to install into. Defaults to the per-user data directory; the maintainer scripts point it at the package tree to regenerate the one table that ships.

Returns:

Where the table landed and what its manifest says.

lost_years.update.status(name: str, *, check_upstream: bool = True) → SourceStatus

Report what is installed for one source and whether it is current.

Parameters:

• name – Source name.

• check_upstream – Ask upstream what it is publishing now. Set False for an offline report of what is installed.

Returns:

The report.

lost_years.update.table_path(name: str) → Path

Return the table a lookup for this source would read.

Parameters:

name – Source name.

Returns:

Path to the Parquet file.

The contract every fetchable life-table source implements.

An update is always download to a scratch directory, build a typed table from what arrived, check it, and only then swap it into place. The check is the point: an upstream that truncates a file, changes a column, or ships a table that disagrees with the published figures must not become the user’s data just because the download completed.

exception lost_years.sources.base.SourceUnavailableError

Upstream could not be reached or refused the request.

class lost_years.sources.base.Source

One upstream life-table database and how to turn it into a table.

name: str

title: str

home_url: str

download_url: str

license: str

filename: str

min_rows: int

ships_in_wheel: bool = False

abstractmethod schema() → Schema

Return the Arrow schema the derived table must conform to.

Returns:

The declared schema, without metadata.

abstractmethod fetch(workdir: Path) → Path

Download the raw upstream artifact.

Parameters:

workdir – Scratch directory to download into.

Returns:

Path to the downloaded artifact.

abstractmethod build(raw: Path, destination: Path) → dict[str, Any]

Turn a raw artifact into the derived Parquet table.

Parameters:

• raw – Downloaded or locally supplied artifact.

• destination – Path to write the Parquet file to.

Returns:

Build notes for the manifest.

abstractmethod release_of(raw: Path) → str

Read the upstream release identifier out of a raw artifact.

Parameters:

raw – Downloaded or locally supplied artifact.

Returns:

Upstream’s own identifier for the release, such as a date.

abstractmethod upstream_release() → str

Ask upstream what its current release is.

Returns:

Upstream’s identifier for the release available right now.

url_for(release: str) → str

Return the URL a given release is published at.

Parameters:

release – Upstream release identifier.

Returns:

The canonical download URL, which for most sources does not depend on the release.

abstractmethod check_values(table: Table) → None

Check the values in a candidate table against known-good figures.

Parameters:

table – The candidate table.

validate(path: Path) → None

Refuse a candidate table that fails schema, size or value checks.

Parameters:

path – Candidate Parquet file.

Raises:

ValidationError – When the table does not conform.

download(url: str, target: Path) → Path

Stream a URL to disk, refusing a short read.

Content-Length is the only cheap end-to-end check a plain HTTP download offers, so a transfer that stops early is caught here rather than surfacing as a corrupt archive later.

Parameters:

• url – Source URL.

• target – Destination path.

Returns:

The destination path.

Raises:

SourceUnavailableError – When upstream refuses, or the transfer is short.

get_text(url: str) → str

Fetch a small text resource.

Parameters:

url – Source URL.

Returns:

The response body.

Raises:

SourceUnavailableError – When upstream refuses or cannot be reached.

Utilities

Shared helpers for reading input frames and matching to life-table rows.

lost_years.utils.isstring(s: Any) → bool

Report whether s is a string.

Parameters:

s – Value to test.

Returns:

True when s is a str.

lost_years.utils.column_exists(df: DataFrame, col: str | None) → bool

Check the column name exists in the DataFrame.

Parameters:

• df – Pandas DataFrame.

• col – Column name.

Returns:

True if exists, False if not exists.

Return type:

bool

lost_years.utils.fixup_columns(cols: list[Any]) → list[str]

Replace index location column to name with col prefix.

Parameters:

cols – List of original columns

Returns:

List of column names

lost_years.utils.closest(lst: list[float] | npt.NDArray[np.floating[Any]], c: float, tolerance: float | None = None) → float

Find closest value in list or array.

A missing target is rejected rather than matched. abs(x - nan) is nan and nan compares False against everything, so min used to fall through to the first element: a row with no age silently returned the life expectancy at age 0.

tolerance bounds how far the answer may sit from the question. Without it, asking for the year 1900 or 2500 against a table that holds only 2022 returns the 2022 figure with nothing to say it is not an answer.

Parameters:

• lst – List of floats or numpy array

• c – Target value to find closest match for

• tolerance – Largest accepted distance between c and the match. None accepts any distance.

Returns:

Closest value in the list/array

Raises:

ValueError – If c is missing, if there is no non-missing candidate, or if the closest candidate is further than tolerance away.

lost_years.utils.download_file(url: str, local_path: str | Path | None = None) → None

Stream url to disk.

Parameters:

• url – Source URL.

• local_path – Destination path. Defaults to the URL’s last path segment.