Metadata

Variable metadata parsing and enrichment functionality.

Schema

Core data structures for representing variables and codebooks.

Pydantic models for metadata representation.

This module defines the core data structures for variables and codebooks, providing type-safe, validated models with rich metadata support.

class statqa.metadata.schema.VariableType(*values)

Bases: StrEnum

Statistical type of a variable.

NUMERIC_CONTINUOUS = 'numeric_continuous'

NUMERIC_DISCRETE = 'numeric_discrete'

CATEGORICAL_NOMINAL = 'categorical_nominal'

CATEGORICAL_ORDINAL = 'categorical_ordinal'

DATETIME = 'datetime'

TEXT = 'text'

BOOLEAN = 'boolean'

UNKNOWN = 'unknown'

class statqa.metadata.schema.DataGeneratingProcess(*values)

Bases: StrEnum

How the data was generated.

OBSERVATIONAL = 'observational'

EXPERIMENTAL = 'experimental'

QUASI_EXPERIMENTAL = 'quasi_experimental'

SURVEY = 'survey'

ADMINISTRATIVE = 'administrative'

SIMULATION = 'simulation'

UNKNOWN = 'unknown'

class statqa.metadata.schema.MissingPattern(*values)

Bases: StrEnum

Pattern of missing data.

MCAR = 'mcar'

MAR = 'mar'

MNAR = 'mnar'

UNKNOWN = 'unknown'

class statqa.metadata.schema.Variable(*, name, label, var_type=VariableType.UNKNOWN, dtype=None, description=None, valid_values=<factory>, missing_values=<factory>, missing_pattern=MissingPattern.UNKNOWN, units=None, range_min=None, range_max=None, is_ordinal=False, dgp=DataGeneratingProcess.UNKNOWN, is_treatment=False, is_outcome=False, is_confounder=False, temporal_variable=None, notes=None, source=None, enriched_metadata=<factory>)

Bases: BaseModel

Represents a single variable/column in a dataset.

Variables:

• name (str) – Variable identifier (e.g., ‘VCF0101’, ‘age’, ‘income’)

• label (str) – Human-readable label/description

• var_type (statqa.metadata.schema.VariableType) – Statistical type of the variable

• dtype (str | None) – Raw data type (from pandas/numpy)

• description (str | None) – Detailed description of what this variable measures

• valid_values (dict[int | str, str]) – Mapping of codes to descriptions (e.g., {1: “Male”, 2: “Female”})

• missing_values (set[int | str]) – Set of codes representing missing data (e.g., {-1, 999})

• missing_pattern (statqa.metadata.schema.MissingPattern) – Pattern of missingness

• units (str | None) – Measurement units (e.g., “years”, “USD”, “percentage”)

• range_min (float | None) – Minimum valid value (for numeric)

• range_max (float | None) – Maximum valid value (for numeric)

• is_ordinal (bool) – Whether categorical variable has meaningful order

• dgp (statqa.metadata.schema.DataGeneratingProcess) – Data generating process

• is_treatment (bool) – Whether this is a treatment/intervention variable

• is_outcome (bool) – Whether this is an outcome/dependent variable

• is_confounder (bool) – Whether this is a potential confounder

• temporal_variable (str | None) – Name of associated time variable (if longitudinal)

• notes (str | None) – Additional metadata notes

• source (str | None) – Data source or survey question text

• enriched_metadata (dict[str, Any]) – LLM-generated enrichment information

Parameters:

• name (str)

• label (str)

• var_type (VariableType)

• dtype (str | None)

• description (str | None)

• valid_values (dict[int | str, str])

• missing_values (set[int | str])

• missing_pattern (MissingPattern)

• units (str | None)

• range_min (float | None)

• range_max (float | None)

• is_ordinal (bool)

• dgp (DataGeneratingProcess)

• is_treatment (bool)

• is_outcome (bool)

• is_confounder (bool)

• temporal_variable (str | None)

• notes (str | None)

• source (str | None)

• enriched_metadata (dict[str, Any])

name: str

label: str

var_type: VariableType

dtype: str | None

description: str | None

valid_values: dict[int | str, str]

missing_values: set[int | str]

missing_pattern: MissingPattern

units: str | None

range_min: float | None

range_max: float | None

is_ordinal: bool

dgp: DataGeneratingProcess

is_treatment: bool

is_outcome: bool

is_confounder: bool

temporal_variable: str | None

notes: str | None

source: str | None

enriched_metadata: dict[str, Any]

classmethod ensure_set(v)

Ensure missing_values is a set.

Parameters:

v (Any)

Return type:

set[int | str]

is_numeric()

Check if variable is numeric.

Return type:

bool

is_categorical()

Check if variable is categorical.

Return type:

bool

is_temporal()

Check if variable represents time.

Return type:

bool

get_cleaned_values()

Get valid values excluding missing codes.

Return type:

dict[int | str, str]

model_config = {'use_enum_values': True, 'validate_assignment': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class statqa.metadata.schema.Codebook(*, name, description=None, variables=<factory>, dataset_info=<factory>, citation=None, version=None, last_updated=None)

Bases: BaseModel

Represents a complete codebook/data dictionary.

Variables:

• name (str) – Codebook name/identifier

• description (str | None) – Overall dataset description

• variables (dict[str, statqa.metadata.schema.Variable]) – Mapping of variable names to Variable objects

• dataset_info (dict[str, Any]) – General dataset metadata

• citation (str | None) – How to cite this dataset

• version (str | None) – Codebook version

• last_updated (str | None) – Last update date

Parameters:

• name (str)

• description (str | None)

• variables (dict[str, Variable])

• dataset_info (dict[str, Any])

• citation (str | None)

• version (str | None)

• last_updated (str | None)

name: str

description: str | None

variables: dict[str, Variable]

dataset_info: dict[str, Any]

citation: str | None

version: str | None

last_updated: str | None

get_variable(name)

Get variable by name.

Parameters:

name (str)

Return type:

Variable | None

get_numeric_variables()

Get all numeric variables.

Return type:

list[Variable]

get_categorical_variables()

Get all categorical variables.

Return type:

list[Variable]

get_temporal_variables()

Get all temporal variables.

Return type:

list[Variable]

get_treatment_variables()

Get all treatment variables.

Return type:

list[Variable]

get_outcome_variables()

Get all outcome variables.

Return type:

list[Variable]

add_variable(variable)

Add a variable to the codebook.

Parameters:

variable (Variable)

Return type:

None

classmethod from_dict(data, name='codebook')

Build a codebook from a parsed JSON mapping, accepting either shape.

A full codebook carries its own metadata and nests the variables under a variables key. Exports written straight from a variable mapping – which is what the bundled example codebooks contain – are a bare {variable_name: {…}} map with no surrounding metadata. Both are accepted so that either can be handed to the CLI.

Parameters:

• data (dict[str, Any]) – Parsed JSON mapping in either shape.

• name (str) – Name to use when the mapping does not carry one of its own.

Returns:

The constructed codebook.

Raises:

ValueError – If data is not a mapping.

Return type:

Codebook

model_config = {'validate_assignment': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

Parsers

Base Parser

Base parser interface for codebook parsing.

Defines the abstract interface that all codebook parsers must implement.

class statqa.metadata.parsers.base.BaseParser(**kwargs)

Bases: ABC

Abstract base class for codebook parsers.

Parameters:

kwargs (Any)

abstractmethod parse(source)

Parse a codebook from the given source.

Parameters:

source (str | Path) – Path to codebook file or string content

Returns:

Parsed Codebook object

Raises:

• ValueError – If source format is invalid

• FileNotFoundError – If source file doesn’t exist

Return type:

Codebook

abstractmethod validate(source)

Check if this parser can handle the given source.

Parameters:

source (str | Path) – Path to codebook file or string content

Returns:

True if parser can handle this source

Return type:

bool

parse_file(file_path)

Convenience method to parse from file path.

Parameters:

file_path (str | Path) – Path to codebook file

Returns:

Parsed Codebook object

Return type:

Codebook

parse_string(content)

Convenience method to parse from string content.

Parameters:

content (str) – Codebook content as string

Returns:

Parsed Codebook object

Return type:

Codebook

CSV Parser

CSV-based codebook parser.

Parses codebooks stored in CSV format with columns like: - variable_name - label - type - description - valid_values - missing_values - units - etc.

class statqa.metadata.parsers.csv.CSVParser(**kwargs)

Bases: BaseParser

Parser for CSV codebooks.

Parameters:

kwargs (Any)

validate(source)

Check if source is valid CSV.

Parameters:

source (str | Path)

Return type:

bool

parse(source)

Parse CSV codebook.

Parameters:

source (str | Path)

Return type:

Codebook

Text Parser

Text-based codebook parser.

Parses structured text codebooks with variable definitions. Supports formats like:

# Variable: age
Label: Respondent Age
Type: numeric_continuous
Units: years
Range: 18-99
Missing: -1, 999
Description: Age of respondent at time of survey

# Variable: gender
Label: Gender
Type: categorical_nominal
Values:
  1: Male
  2: Female
  3: Other
Missing: 0

class statqa.metadata.parsers.text.TextParser(**kwargs)

Bases: BaseParser

Parser for structured text codebooks.

Parameters:

kwargs (Any)

validate(source)

Check if source is valid text format.

Parameters:

source (str | Path)

Return type:

bool

parse(source)

Parse text codebook.

Parameters:

source (str | Path)

Return type:

Codebook

Statistical Formats Parser

Statistical format parser for SPSS, Stata, and SAS files.

Uses pyreadstat library to parse statistical data files and extract rich metadata including variable labels, value labels, and missing value definitions.

class statqa.metadata.parsers.statistical.StatisticalFormatParser(**kwargs)

Bases: BaseParser

Parser for statistical data files (SPSS, Stata, SAS).

Parameters:

kwargs (Any)

validate(source)

Check if source is a supported statistical format.

Parameters:

source (str | Path)

Return type:

bool

parse(source)

Parse statistical format file.

Parameters:

source (str | Path)

Return type:

Codebook

Enricher

LLM-powered metadata enhancement.

LLM-based metadata enrichment.

Uses language models to verify, infer, and enrich variable metadata including: - Type inference and validation - Relationship suggestions - Causal structure hints - Missing pattern detection - Variable importance ranking

class statqa.metadata.enricher.MetadataEnricher(provider='openai', model=None, api_key=None, **kwargs)

Bases: object

Enrich metadata using LLM capabilities.

Supports both OpenAI and Anthropic models.

Parameters:

• provider (Literal['openai', 'anthropic'])

• model (str | None)

• api_key (str | None)

• kwargs (Any)

client: Any

enrich_variable(variable, dataset_context=None)

Enrich a single variable’s metadata.

Parameters:

• variable (Variable) – Variable to enrich

• dataset_context (str | None) – Optional context about the dataset

Returns:

Enriched Variable with updated metadata

Raises:

• EnrichmentError – If enrichment process fails

• LLMConnectionError – If LLM connection fails

• LLMResponseError – If LLM response is invalid

Return type:

Variable

enrich_codebook(codebook)

Enrich entire codebook metadata.

Parameters:

codebook (Codebook) – Codebook to enrich

Returns:

Enriched Codebook

Return type:

Codebook