piinata finds likely personal data in research files and reports where it is, with the evidence, so a person can decide what to do. This page explains the design: why it exists, how a scan proceeds, what it reads and detects, and how we check that it works.
Why another scanner
Before a research dataset is shared, someone has to find the personal
data in it. The existing free scanners do not do this well. Measured on
a synthetic survey with known answers (the benchmark in
data-raw/benchmark/ of the repository, 28 columns, 12
personal):
| Tool | Reads | Looks at | Result |
|---|---|---|---|
J-PAL PII-Scan (R, last commit 2022) |
csv, dta, sav, sas7bdat | 56 substrings in column names and labels | 6 of 12 personal columns; 12 of 16 keyword-trap columns flagged;
does not start under current optparse;
--strict flags nothing |
J-PAL stata_PII_scan
|
dta | the same list, plus every string longer than 3 characters | not run (no Stata); by its rules it flags nearly every text column |
IPA PII_detection
|
csv, xlsx, dta | word lists, three value regexes, uniqueness, a Google population lookup | not run; a Windows desktop application, not a library |
detector (CRAN, 2015) |
data frames | email, phone, national id | abandoned |
Two failures recur. The tools skip files they cannot open without
saying so, and they judge a column by its name, so an email address in a
column called q12 is invisible.
Principles
- Read first. Nothing is scanned until it has been read, and every file the scan was pointed at is accounted for: read, unsupported, or failed, with the reason. A file that could not be read is a finding, not a silence.
- Do not hand-roll. Reading, parsing and validation come from maintained CRAN packages. Where none exists the gap is recorded here and the feature waits, is optional, or is delegated to a backend.
- Three kinds of evidence. What a field is called (name, label, question text); what it contains (values); how it behaves (uniqueness, free text). A finding says which kind it rests on.
-
Quiet by default. A finding needs a share of
matching values, not one stray hit, and names are matched on whole
words. Any field, type of finding or file can be allowed, with a reason,
in
.piinata.yml. - The scanner must not leak. Example values in reports are masked unless asked for. The core makes no network calls.
-
A small core, optional backends. The core installs
with no Python, no Java and no system libraries beyond what its imports
need. Heavier detectors (Presidio, named-entity models) are opt-in, and
the sister packages
idcheckanddialtoneare used when installed.
Pipeline
A scan is four stages, each in its own files, each replaceable:
| Stage | Does | Code |
|---|---|---|
| read | turns files and objects into fields; accounts for every file |
R/read.R, R/readers.R,
R/fields.R
|
| normalize | cleans values so patterns can match dirty data | R/normalize.R |
| detect | name, value and structure evidence, one function each |
R/terms.R, R/detect_name.R,
R/detect_value.R, R/detect_custom.R,
R/detect_table.R
|
| report | findings, summary, the check that gates a script |
R/findings.R, R/scan.R,
R/report.R
|
Two more stages act on findings: decide (R/config.R,
R/review.R), where a person allows or removes each flagged
field and the decision is recorded in .piinata.yml, and
redact (R/redact.R, R/redact_files.R), which
drops, masks or pseudonymizes what was flagged, in a data frame or in a
copy of each file written beside the original, and can keep the
originals out of git (R/gitignore.R).
Normalization exists because real exports are dirty. It applies
Unicode compatibility normalization, turns decimal digits of any script
into ASCII digits, drops zero-width and other invisible format
characters, and collapses whitespace, all through ICU
(stringi). It does not yet undo deliberate obfuscation
(“name at gmail dot com”), spelled-out digits, or OCR confusions such as
O for 0; those belong to an optional, more aggressive cleaning step.
The flagger proposes and a person disposes. A finding is a proposal with its evidence; the decision (keep, drop, redact, allow with a reason) is the user’s, is recorded in the project, and is what the check enforces.
Reading
Everything is read into one shape, a table of fields:
| column | meaning |
|---|---|
source |
file path, or the name of the object passed in |
locator |
where in the source: column, sheet and column, page, line range |
name |
column name, or NA for running text |
label |
variable label or question text, when the format carries one |
values |
list-column of character vectors |
kind |
tabular or text
|
A table gives one field per column. A document gives one field per page or paragraph block. Detectors only ever see fields, so a detector written once works on a Stata file, a PDF and a data frame alike.
Readers are the tidyverse and r-lib packages social scientists already use:
| Input | Reader | Notes |
|---|---|---|
| csv, tsv, psv | readr |
everything read as text |
| xlsx, xls | readxl |
every sheet |
| dta, sav, zsav, por, sas7bdat, xpt | haven |
variable and value labels kept |
| rds | base R | data frames and character vectors only |
| txt, md, log, do, R, tex and similar | readr::read_lines() |
scripts and logs can list rows of data |
| Qualtrics csv/tsv export | qualtRics::read_survey() |
detected from its ImportId row; question text becomes
the label |
| parquet | nanoparquet |
|
| json, jsonl, ndjson | jsonlite |
tabular JSON as a table, otherwise as text |
pdftools |
text layer only; a scan is reported as unsupported | |
| docx | officer |
|
| rtf | striprtf |
|
| html | rvest |
|
| zip, tar, tgz, gz, bz2, xz | base R (unzip, untar, connections) |
members read like files and named archive!member;
nested three deep; over 2 GB unpacked is reported |
| ipynb | jsonlite |
cell source and printed outputs, one field per cell |
| geojson, kml, kmz |
jsonlite, xml2
|
attributes, plus each feature’s first coordinate pair as
longitude and latitude
|
| shp, dbf, gpkg |
sf, foreign
|
attributes from the .dbf; geometry as a point on each shape, in degrees; side files ignored |
| sqlite, db | RSQLite |
every table; a .db that is not SQLite is unsupported |
| ods, xlsm, pptx |
readODS, readxl, officer
|
every sheet; one field per slide |
| vcf, eml, mbox, ics, srt, vtt, and scripts | readr::read_lines() |
contacts, mail, calendars and transcripts are mostly personal data |
| data frame, character vector | none | passed straight through |
readr, readxl and haven are
imports. The rest are suggested and checked at run time: a missing
package produces an “unsupported: install X” row in the read report, not
an error.
Still to read, in the order social scientists meet them: SurveyCTO and KoboToolbox/ODK exports with their form definitions (question text as labels, repeat groups, media file names), REDCap exports with their data dictionary, SurveyMonkey and Google Forms headers, and the file names themselves, which often carry a respondent’s name. Legacy .doc files need an external converter and are reported as unsupported.
Hidden files are read: .Rhistory and .env
files hold personal data as often as anything. Folders tools keep for
themselves (.git, .venv,
node_modules, .Rproj.user, caches) are
reported once each as ignored, and macOS’s .DS_Store files
are reported as ignored.
Not in the first release: images and scanned PDFs (would use
tesseract), Outlook .pst and .msg
files, live databases, and live survey-platform APIs
(rsurveycto, robotoolbox exist and can be
added as readers later).
Detection
A detector is a function from a field to zero or more findings. The core set:
| Evidence | Detector | Built on |
|---|---|---|
| name | terms for names, contact, birth, location, identifiers, relatives,
in English with some Spanish, French and Hindi; matched on whole tokens
after splitting snake_case, camelCase and
digits |
stringr |
| name | platform fields: Qualtrics (IPAddress,
RecipientEmail, LocationLatitude …), SurveyCTO
(deviceid, subscriberid,
devicephonenum …) |
fixed lists |
| value | email address | stringr |
| value | IP address |
ipaddress for validation |
| value | date of birth: parseable dates in a plausible birth range | lubridate |
| table | coordinates: two neighboring columns of numbers in range with four or more decimals, at least one within ±90; a column named as a rate, price, score or the like is never one | base R |
| value | addresses: two of a postcode, a house marker and an area word
(street, avenue, ward, tehsil …); postcodes for the project’s
regions
|
Google’s address metadata, stored with the package |
| value | identity numbers: Aadhaar, PAN, GSTIN, voter id, passport, US SSN, UK NHS, payment cards | idcheck |
| value | IBANs, MAC addresses, UUIDs | checkstring |
| context | a value finding moves up a grade when the column’s name or label names its kind, as Presidio raises a match with a context word nearby | context_terms() |
| value | phone numbers: 10 to 13 digits, graded by how they are written and
how much of a column they fill; checked against libphonenumber’s
numbering plans for the project’s regions
|
dialtone, when installed |
| structure | identifier-like: nearly all values distinct | base R |
| structure | free text: long, varied strings; value detectors then run cell by cell | base R |
A value finding carries the number and share of matching values. The default threshold is 5% of non-missing values or 5 values, whichever is larger; for free-text fields any match is reported.
Optional backends, each behind one function and one suggested package:
| Backend | Adds | Requires |
|---|---|---|
use_gliner() |
a second look at columns the rules leave uncertain: names behind question numbers, names in lower case or Devanagari, place names the heuristic takes for people; votes on whole columns, never decides a cell |
reticulate; the Python gliner package and
the Apache-2.0 urchade/gliner_multi_pii-v1 model, declared
with py_require()
|
Presidio is not a backend: its pattern recognizers repeat what the
core, idcheck and dialtone do, its name and
place detection is whichever model it is given, and its table module
labels a column after a single hit. It is run once, as a comparison.
nametagger is not used: its only English model dates from
2014 and is licensed for non-commercial use.
Output
pii_scan() returns a tibble, one row per finding:
source, locator, name,
label, type (email, phone, person_name …),
evidence (name, value, structure), detector,
n, share, examples (masked),
confidence (low, medium, high).
Around it:
-
pii_read(path)returns the fields and, as an attribute printed with them, the read report. -
pii_summary(findings)gives one row per field with the strongest evidence. -
pii_check(findings, allow = )errors when anything unallowed remains, so a script or CI job can gate on it. - A project’s own identifiers are declared as patterns in
.piinata.yml(seepii_config()): a regex or a word list, with optional context words, exclusions and a check-digit validation.
Function names share the pii_ prefix, except where R has
a convention of its own: as_pii_fields() converts, as
as_*() functions do, and use_gliner() sets up
a backend, as use_*() functions do. Arguments are data
first.
A finding’s type is the kind of personal data, from a
small vocabulary that does not change with the rule that found it:
person_name, phone, email,
address, date_of_birth, location,
coordinates, government_id,
financial_account, ip_address,
online_id, device_id, uuid, and
for structure free_text and unique_text, plus
the types of a project’s own patterns. Which rule matched is in
detector: idcheck:us_ssn,
checkstring:iban, gliner,
custom:study_id.
Speed
A scan must be cheap enough to run on every commit.
- Detectors see the distinct values of a column and their counts, so a coded column costs the same at a thousand rows or a million.
- Column types are used: logical columns are skipped, numeric and date columns skip the text patterns.
- Above
sampledistinct values (default 10,000) a column is judged from an evenly spaced subset.sample = Infexamines everything. Running text is always examined in full. - Date formats are only guessed for values shaped like dates. Guessing on arbitrary strings took 336 seconds on one real column; it now takes none.
- Measured: a real project folder of 50 files and 3,137 fields scans
in about 15 seconds on a laptop. The sister package’s checksums are C++
(
cpp11). - Later, where profiling says so: parallel scanning of files, and an exhaustive mode for ragged sources such as YAML, nested JSON and long text.
How we know it works
-
Benchmark, tabular. The benchmark in
data-raw/benchmark/of the repository holds 20 personal columns and 26 others, with Indian and US names and addresses, names in lower case and Devanagari behind question numbers, Social Security, card and IBAN numbers, and decoys: village, facility, scheme, caste, crop, city, business and street names, and nine-digit order numbers.score_piinata.Rscores a scan at themediumlevel thatpii_check()gates on and atlow. The core catches 15 of 20 atmediumwith no false alarms, and 18 of 20 atlowwith two (village names read as person names; order numbers shaped like SSNs, which no rule can tell from SSNs without a column name). Withuse_gliner()it catches 18 of 20 atmediumwith no false alarms, missing the names in Devanagari and the SSNs behind a question number (score_gliner.R; the 70% cut formediumwas set on SOTAB’s validation split, below). The Qualtrics export: 6 of 9 atmedium, 9 of 9 atlow. The same file in csv, dta, xlsx, parquet, tsv and ndjson must give the same findings. - Benchmark, text. A txt, a docx and a pdf with planted emails, phones and dates: every planted item found, located to page or line.
- Read report. A folder holding one unreadable file and one unsupported file yields two rows saying so, and the scan still completes.
-
An outside benchmark. SOTAB’s real web tables,
scored as described in
data-raw/benchmark/external/README.md: tuned on validation, scored once on test. On test piinata finds 106 of 217 personal columns with 80 false alarms among 1,582 (F1 0.53), and 155 with 86 withuse_gliner()(F1 0.68). Presidio’s best configurations reach F1 0.42 (GLiNER, organizations not counted: 137 found, 293 false alarms) and 0.41 (spaCy, locations not counted: 81 found, 99 false alarms). The validation split taught most of what the synthetic benchmark could not: obfuscated emails, dotted phones, year spans, ordinals, digits in URLs, address cues far apart, and that a column of past dates is not evidence of birth dates. -
A second opinion on the synthetic benchmark.
Presidio 2.2.366 with its default English pipeline (spaCy
en_core_web_lg), run on the same benchmark byscore_presidio.py. Judged as presidio-structured judges, by any detection in a column, it catches 18 of 20 personal columns with 10 false alarms among 26; requiring half the rows, 13 of 20 with 4. Its errors are instructive. spaCy reads crop varieties (86% of rows), village names (81%) and scheme names (59%) as people. Its phone recognizer scores a match 0.4, below the usual 0.5 cut, and relies on a word such as “phone” within five words before the number to add 0.35, so in a table cell, where no such word is near and the column’s name is not passed, no phone number is found;+91numbers come out as dates. It sees 3% of the Devanagari names. - Real data. Scans of real project folders, compared by hand. The first, a folder of election and survey files, cut flagged fields from 465 to 251 by showing where the rules were loose (place names read as person names, precise numbers as coordinates, keys as text), and found a column of real mobile numbers. The benchmark and the detectors were written by the same hand, so real folders are the check that counts.
-
R CMD check --as-cranclean on the five r-canon platforms with no optional backend installed. Examples and tests run offline in seconds.
The benchmark is adversarial by construction, so its false-alarm rate is a stress test, not an estimate for typical data.