Skip to contents

piinata finds likely personal data in research files and reports where it is, with the evidence, so a person can decide what to do. This page explains the design: why it exists, how a scan proceeds, what it reads and detects, and how we check that it works.

Why another scanner

Before a research dataset is shared, someone has to find the personal data in it. The existing free scanners do not do this well. Measured on a synthetic survey with known answers (the benchmark in data-raw/benchmark/ of the repository, 28 columns, 12 personal):

Tool Reads Looks at Result
J-PAL PII-Scan (R, last commit 2022) csv, dta, sav, sas7bdat 56 substrings in column names and labels 6 of 12 personal columns; 12 of 16 keyword-trap columns flagged; does not start under current optparse; --strict flags nothing
J-PAL stata_PII_scan dta the same list, plus every string longer than 3 characters not run (no Stata); by its rules it flags nearly every text column
IPA PII_detection csv, xlsx, dta word lists, three value regexes, uniqueness, a Google population lookup not run; a Windows desktop application, not a library
detector (CRAN, 2015) data frames email, phone, national id abandoned

Two failures recur. The tools skip files they cannot open without saying so, and they judge a column by its name, so an email address in a column called q12 is invisible.

Principles

  1. Read first. Nothing is scanned until it has been read, and every file the scan was pointed at is accounted for: read, unsupported, or failed, with the reason. A file that could not be read is a finding, not a silence.
  2. Do not hand-roll. Reading, parsing and validation come from maintained CRAN packages. Where none exists the gap is recorded here and the feature waits, is optional, or is delegated to a backend.
  3. Three kinds of evidence. What a field is called (name, label, question text); what it contains (values); how it behaves (uniqueness, free text). A finding says which kind it rests on.
  4. Quiet by default. A finding needs a share of matching values, not one stray hit, and names are matched on whole words. Any field, type of finding or file can be allowed, with a reason, in .piinata.yml.
  5. The scanner must not leak. Example values in reports are masked unless asked for. The core makes no network calls.
  6. A small core, optional backends. The core installs with no Python, no Java and no system libraries beyond what its imports need. Heavier detectors (Presidio, named-entity models) are opt-in, and the sister packages idcheck and dialtone are used when installed.

Pipeline

A scan is four stages, each in its own files, each replaceable:

Stage Does Code
read turns files and objects into fields; accounts for every file R/read.R, R/readers.R, R/fields.R
normalize cleans values so patterns can match dirty data R/normalize.R
detect name, value and structure evidence, one function each R/terms.R, R/detect_name.R, R/detect_value.R, R/detect_custom.R, R/detect_table.R
report findings, summary, the check that gates a script R/findings.R, R/scan.R, R/report.R

Two more stages act on findings: decide (R/config.R, R/review.R), where a person allows or removes each flagged field and the decision is recorded in .piinata.yml, and redact (R/redact.R, R/redact_files.R), which drops, masks or pseudonymizes what was flagged, in a data frame or in a copy of each file written beside the original, and can keep the originals out of git (R/gitignore.R).

Normalization exists because real exports are dirty. It applies Unicode compatibility normalization, turns decimal digits of any script into ASCII digits, drops zero-width and other invisible format characters, and collapses whitespace, all through ICU (stringi). It does not yet undo deliberate obfuscation (“name at gmail dot com”), spelled-out digits, or OCR confusions such as O for 0; those belong to an optional, more aggressive cleaning step.

The flagger proposes and a person disposes. A finding is a proposal with its evidence; the decision (keep, drop, redact, allow with a reason) is the user’s, is recorded in the project, and is what the check enforces.

Reading

Everything is read into one shape, a table of fields:

column meaning
source file path, or the name of the object passed in
locator where in the source: column, sheet and column, page, line range
name column name, or NA for running text
label variable label or question text, when the format carries one
values list-column of character vectors
kind tabular or text

A table gives one field per column. A document gives one field per page or paragraph block. Detectors only ever see fields, so a detector written once works on a Stata file, a PDF and a data frame alike.

Readers are the tidyverse and r-lib packages social scientists already use:

Input Reader Notes
csv, tsv, psv readr everything read as text
xlsx, xls readxl every sheet
dta, sav, zsav, por, sas7bdat, xpt haven variable and value labels kept
rds base R data frames and character vectors only
txt, md, log, do, R, tex and similar readr::read_lines() scripts and logs can list rows of data
Qualtrics csv/tsv export qualtRics::read_survey() detected from its ImportId row; question text becomes the label
parquet nanoparquet
json, jsonl, ndjson jsonlite tabular JSON as a table, otherwise as text
pdf pdftools text layer only; a scan is reported as unsupported
docx officer
rtf striprtf
html rvest
zip, tar, tgz, gz, bz2, xz base R (unzip, untar, connections) members read like files and named archive!member; nested three deep; over 2 GB unpacked is reported
ipynb jsonlite cell source and printed outputs, one field per cell
geojson, kml, kmz jsonlite, xml2 attributes, plus each feature’s first coordinate pair as longitude and latitude
shp, dbf, gpkg sf, foreign attributes from the .dbf; geometry as a point on each shape, in degrees; side files ignored
sqlite, db RSQLite every table; a .db that is not SQLite is unsupported
ods, xlsm, pptx readODS, readxl, officer every sheet; one field per slide
vcf, eml, mbox, ics, srt, vtt, and scripts readr::read_lines() contacts, mail, calendars and transcripts are mostly personal data
data frame, character vector none passed straight through

readr, readxl and haven are imports. The rest are suggested and checked at run time: a missing package produces an “unsupported: install X” row in the read report, not an error.

Still to read, in the order social scientists meet them: SurveyCTO and KoboToolbox/ODK exports with their form definitions (question text as labels, repeat groups, media file names), REDCap exports with their data dictionary, SurveyMonkey and Google Forms headers, and the file names themselves, which often carry a respondent’s name. Legacy .doc files need an external converter and are reported as unsupported.

Hidden files are read: .Rhistory and .env files hold personal data as often as anything. Folders tools keep for themselves (.git, .venv, node_modules, .Rproj.user, caches) are reported once each as ignored, and macOS’s .DS_Store files are reported as ignored.

Not in the first release: images and scanned PDFs (would use tesseract), Outlook .pst and .msg files, live databases, and live survey-platform APIs (rsurveycto, robotoolbox exist and can be added as readers later).

Detection

A detector is a function from a field to zero or more findings. The core set:

Evidence Detector Built on
name terms for names, contact, birth, location, identifiers, relatives, in English with some Spanish, French and Hindi; matched on whole tokens after splitting snake_case, camelCase and digits stringr
name platform fields: Qualtrics (IPAddress, RecipientEmail, LocationLatitude …), SurveyCTO (deviceid, subscriberid, devicephonenum …) fixed lists
value email address stringr
value IP address ipaddress for validation
value date of birth: parseable dates in a plausible birth range lubridate
table coordinates: two neighboring columns of numbers in range with four or more decimals, at least one within ±90; a column named as a rate, price, score or the like is never one base R
value addresses: two of a postcode, a house marker and an area word (street, avenue, ward, tehsil …); postcodes for the project’s regions Google’s address metadata, stored with the package
value identity numbers: Aadhaar, PAN, GSTIN, voter id, passport, US SSN, UK NHS, payment cards idcheck
value IBANs, MAC addresses, UUIDs checkstring
context a value finding moves up a grade when the column’s name or label names its kind, as Presidio raises a match with a context word nearby context_terms()
value phone numbers: 10 to 13 digits, graded by how they are written and how much of a column they fill; checked against libphonenumber’s numbering plans for the project’s regions dialtone, when installed
structure identifier-like: nearly all values distinct base R
structure free text: long, varied strings; value detectors then run cell by cell base R

A value finding carries the number and share of matching values. The default threshold is 5% of non-missing values or 5 values, whichever is larger; for free-text fields any match is reported.

Optional backends, each behind one function and one suggested package:

Backend Adds Requires
use_gliner() a second look at columns the rules leave uncertain: names behind question numbers, names in lower case or Devanagari, place names the heuristic takes for people; votes on whole columns, never decides a cell reticulate; the Python gliner package and the Apache-2.0 urchade/gliner_multi_pii-v1 model, declared with py_require()

Presidio is not a backend: its pattern recognizers repeat what the core, idcheck and dialtone do, its name and place detection is whichever model it is given, and its table module labels a column after a single hit. It is run once, as a comparison. nametagger is not used: its only English model dates from 2014 and is licensed for non-commercial use.

Output

pii_scan() returns a tibble, one row per finding:

source, locator, name, label, type (email, phone, person_name …), evidence (name, value, structure), detector, n, share, examples (masked), confidence (low, medium, high).

Around it:

  • pii_read(path) returns the fields and, as an attribute printed with them, the read report.
  • pii_summary(findings) gives one row per field with the strongest evidence.
  • pii_check(findings, allow = ) errors when anything unallowed remains, so a script or CI job can gate on it.
  • A project’s own identifiers are declared as patterns in .piinata.yml (see pii_config()): a regex or a word list, with optional context words, exclusions and a check-digit validation.

Function names share the pii_ prefix, except where R has a convention of its own: as_pii_fields() converts, as as_*() functions do, and use_gliner() sets up a backend, as use_*() functions do. Arguments are data first.

A finding’s type is the kind of personal data, from a small vocabulary that does not change with the rule that found it: person_name, phone, email, address, date_of_birth, location, coordinates, government_id, financial_account, ip_address, online_id, device_id, uuid, and for structure free_text and unique_text, plus the types of a project’s own patterns. Which rule matched is in detector: idcheck:us_ssn, checkstring:iban, gliner, custom:study_id.

Speed

A scan must be cheap enough to run on every commit.

  • Detectors see the distinct values of a column and their counts, so a coded column costs the same at a thousand rows or a million.
  • Column types are used: logical columns are skipped, numeric and date columns skip the text patterns.
  • Above sample distinct values (default 10,000) a column is judged from an evenly spaced subset. sample = Inf examines everything. Running text is always examined in full.
  • Date formats are only guessed for values shaped like dates. Guessing on arbitrary strings took 336 seconds on one real column; it now takes none.
  • Measured: a real project folder of 50 files and 3,137 fields scans in about 15 seconds on a laptop. The sister package’s checksums are C++ (cpp11).
  • Later, where profiling says so: parallel scanning of files, and an exhaustive mode for ragged sources such as YAML, nested JSON and long text.

How we know it works

  1. Benchmark, tabular. The benchmark in data-raw/benchmark/ of the repository holds 20 personal columns and 26 others, with Indian and US names and addresses, names in lower case and Devanagari behind question numbers, Social Security, card and IBAN numbers, and decoys: village, facility, scheme, caste, crop, city, business and street names, and nine-digit order numbers. score_piinata.R scores a scan at the medium level that pii_check() gates on and at low. The core catches 15 of 20 at medium with no false alarms, and 18 of 20 at low with two (village names read as person names; order numbers shaped like SSNs, which no rule can tell from SSNs without a column name). With use_gliner() it catches 18 of 20 at medium with no false alarms, missing the names in Devanagari and the SSNs behind a question number (score_gliner.R; the 70% cut for medium was set on SOTAB’s validation split, below). The Qualtrics export: 6 of 9 at medium, 9 of 9 at low. The same file in csv, dta, xlsx, parquet, tsv and ndjson must give the same findings.
  2. Benchmark, text. A txt, a docx and a pdf with planted emails, phones and dates: every planted item found, located to page or line.
  3. Read report. A folder holding one unreadable file and one unsupported file yields two rows saying so, and the scan still completes.
  4. An outside benchmark. SOTAB’s real web tables, scored as described in data-raw/benchmark/external/README.md: tuned on validation, scored once on test. On test piinata finds 106 of 217 personal columns with 80 false alarms among 1,582 (F1 0.53), and 155 with 86 with use_gliner() (F1 0.68). Presidio’s best configurations reach F1 0.42 (GLiNER, organizations not counted: 137 found, 293 false alarms) and 0.41 (spaCy, locations not counted: 81 found, 99 false alarms). The validation split taught most of what the synthetic benchmark could not: obfuscated emails, dotted phones, year spans, ordinals, digits in URLs, address cues far apart, and that a column of past dates is not evidence of birth dates.
  5. A second opinion on the synthetic benchmark. Presidio 2.2.366 with its default English pipeline (spaCy en_core_web_lg), run on the same benchmark by score_presidio.py. Judged as presidio-structured judges, by any detection in a column, it catches 18 of 20 personal columns with 10 false alarms among 26; requiring half the rows, 13 of 20 with 4. Its errors are instructive. spaCy reads crop varieties (86% of rows), village names (81%) and scheme names (59%) as people. Its phone recognizer scores a match 0.4, below the usual 0.5 cut, and relies on a word such as “phone” within five words before the number to add 0.35, so in a table cell, where no such word is near and the column’s name is not passed, no phone number is found; +91 numbers come out as dates. It sees 3% of the Devanagari names.
  6. Real data. Scans of real project folders, compared by hand. The first, a folder of election and survey files, cut flagged fields from 465 to 251 by showing where the rules were loose (place names read as person names, precise numbers as coordinates, keys as text), and found a column of real mobile numbers. The benchmark and the detectors were written by the same hand, so real folders are the check that counts.
  7. R CMD check --as-cran clean on the five r-canon platforms with no optional backend installed. Examples and tests run offline in seconds.

The benchmark is adversarial by construction, so its false-alarm rate is a stress test, not an estimate for typical data.

Out of scope

  • Indirect identification from combinations of traits. sdcMicro does this.
  • Secrets such as API keys. gitleaks does this.
  • Images, audio, scanned documents.