Benchmark

How we measure Obscate’s accuracy

We publish our evaluation method and numbers because an unverifiable accuracy claim is worthless to a security team. Everything below is reproducible from the eval harness (eval/) with one command.

What we test

  1. Detection — does Obscate find the sensitive data? (precision, recall, F1)
  2. Restoration — after redact → restore, is the output identical to the input? (exact-match rate, per reversible strategy)
  3. Latency — how long redaction and a full round trip take per document.

The dataset

Read this caveat before the numbers. The structured values are generated by Obscate’s own synthetic-value module, so they are well-formed by construction, and the templates are short and clean (80 characters on average). That makes the structured-ID results an upper bound for clean input, not a claim about messy real-world text. We’ll publish results on additional corpora as we build them.

Metrics, defined

Results

Detection

EntityPrecisionRecallF1Spans
Aadhaar1.001.001.0029
Credit card1.001.001.0037
Email1.001.001.0057
GSTIN1.001.001.0016
IBAN1.001.001.0024
IPv41.001.001.0021
MAC address1.001.001.0021
PAN1.001.001.0029
Phone1.001.001.0056
US SSN1.001.001.0014
Person0.790.800.80137
Location0.770.770.7743
Micro average0.920.920.92484
Macro average F10.96

Micro average: TP 447 · FP 39 · FN 37.

Why the structured IDs score 1.00. Cards (Luhn), Aadhaar (Verhoeff), IBAN (mod-97) and GSTIN (check character) are checksum-validated: a match that fails its checksum is rejected, which is what keeps precision at 1.00. PAN, SSN, email, phone, IPv4 and MAC are pattern-matched without a checksum; they also score 1.00 here, helped by the clean synthetic input (see the caveat).

Restoration

StrategyExact restores
fpe (map-free, key only)200 / 200
tokenize (with token map)200 / 200
synthetic (with token map)200 / 200

Beyond the benchmark, randomized property tests (Hypothesis) check the round-trip on adversarial generated input:

These tests have caught and fixed real bugs before release, for example an IBAN followed by a number that went undetected.

Latency (per document)

StrategyRedact medianRedact p95Round-trip medianRound-trip p95
fpe1.86 ms2.23 ms2.02 ms2.47 ms
tokenize1.39 ms1.56 ms1.39 ms1.57 ms
synthetic1.42 ms1.64 ms1.45 ms1.68 ms

Measured in-process on a single thread with no network, on an Apple M5 laptop (Python 3.12, macOS), over the benchmark’s short documents (80 characters on average), with the spaCy NER model loaded. This is engine time, not API latency: a hosted API call adds network round-trip and server overhead, and longer documents take longer. We’ll publish hosted-API latency once the service is live.

Limitations (stated plainly)

Reproduce it yourself

pip install -e ".[ner]" && python -m spacy download en_core_web_sm
python -m eval.benchmark

The dataset generator (eval/dataset.py) and scorer (eval/benchmark.py) are both in the repo, so you can see exactly what was tested and run it again. Latency figures will differ on other hardware.

Contact

Our contact address is being set up and will be published here before launch.