Skip to contents

What is planned, in the order it is planned, and the questions still open. Changes to this page are proposals; open an issue to argue with one.

Using it in a workflow

Researchers call it from R. Projects also need it in the flow of work:

  1. Review and exceptions. pii_review() writes the findings as a file with a decision column for a person to fill in. A committed .piinata.yml holds what has been reviewed: fields and files that may stay, detectors turned off, thresholds, and extra terms. pii_check() reads it. An exception names a field and a reason, not a blanket switch.
  2. Custom patterns. Users declare their own: a name, a regular expression or a function, an optional validator and a confidence. Study ids, staff codes and local identifiers go here.
  3. Git. A pre-commit hook (language: r in .pre-commit-hooks.yaml) and a GitHub Action that scan the files a commit touches and fail on unreviewed findings. A Python port is on the table if no Python package does the same job, since most hooks run in Python environments.
  4. Redaction. The levels are: report; drop a field; replace values with a placeholder of the same type; replace with a stable pseudonym, so joins and counts still work; and a custom function. The model is Presidio’s split between an analyzer that returns typed spans with scores and an anonymizer that applies an operator per type (replace, mask, hash, redact, encrypt, custom). pii_scan() is the analyzer; pii_redact() is the anonymizer, and pii_redact_files() writes a clean copy beside the original. Still to do: JSON, and a custom function as a method.
  5. Originals out of git. Done: pii_ignore_originals() adds each original to .gitignore so the clean copy can be committed. It is opt-in and reports what it did; it does not rewrite history, so an original already committed stays in the repository’s past.
  6. URLs and browsing data. A URL can carry personal data in its user, path and query string. A URL detector would parse with a maintained parser and report or strip those parts while keeping the host.

Items 1 to 3 come before redaction: a scanner people cannot quiet is one they remove.

Versions

Version Contents State
0.1.0 first release, on GitHub: reading (archives, notebooks, geographic files, databases), name, value and structure detectors, idcheck, dialtone and checkstring, context, addresses, coordinate pairs, review and accept, redaction of data frames and files, originals out of git, use_gliner(), the synthetic and SOTAB benchmarks done
0.2.0 CRAN, once idcheck and dialtone are there
0.3.0 SurveyCTO, KoboToolbox/ODK, REDCap, SurveyMonkey and Google Forms exports with their form definitions; file-name scanning
0.4.0 pre-commit hook and GitHub Action
0.5.0 names in Devanagari
0.6.0 URL parts

Open questions

  1. Phone numbers. Decided and built: no maintained, Java-free libphonenumber existed on CRAN (dialvalidator was archived on 2026-03-30 and dialr needs Java), so the sister package dialtone validates numbers on Google’s libphonenumber metadata (Apache-2.0). piinata uses it when installed: numbers with a country code are always checked, others against the project’s regions; a rejected number stops counting, and a column of valid numbers moves up a grade.
  2. National ids. Decided and built: the sister package idcheck validates identity numbers in C++ (Aadhaar by Verhoeff checksum, GSTIN by checksum, PAN, voter id, passport and Indian mobile numbers by format). piinata uses it when installed; a checksum match is high confidence. CRAN does have two Verhoeff packages, verhoeff and CheckDigit, but both are GPL and neither is built for scanning a column.
  3. Person names in values. Without a model, the core can only say “short, capitalized, mostly distinct strings”, confidence low, and cannot tell a column of names from a column of village names. Decided, after a review of GLiNER, Presidio, spaCy, Flair, Stanza, IndicNER and the R routes: use_gliner() through reticulate::py_require() with the Apache-2.0 urchade/gliner_multi_pii-v1 model, run only on columns the rules leave uncertain and voting on the share of rows, because it manages 2 to 40 cells a second on a CPU. Independent tests put such models at 0.4 to 0.6 F1 on unfamiliar text and none covers short cells, so it is trusted only as far as the benchmark’s decoys show. Not adopted: nametagger (2014 English model, non-commercial license) and Presidio as a backend.
  4. The name. piinata is free on CRAN and GitHub.