Benchmark: detection quality and runtime overhead¶

This page gives the numbers behind Taintward's detection step (taintward/sdk/detect.py): how much sensitive data it finds, how often it raises a false alarm, and how much time it adds to every tool call. Everything here can be re-run from the bench/ folder.

Three rounds of work are recorded here. Round 1 added JWT and private key recognizers. Round 2 (the "fast path") fixed the two problems round 1 found: detection was far too slow, and half of all clean payloads were flagged. Round 3 (gap G9) is about data outside the US formats and is measured on outside data only: see Results on outside data.

Summary¶

	before round 2	after round 2
false alarm rate on hard negatives (payload mode)	0.52	0.00
example-level recall (payload mode)	0.94	0.94
span precision, strict	0.42	0.96
span recall, strict	0.85	0.94
wrapped tool call, 1 KB, p50	518 ms	8.4 ms
wrapped tool call, 1 KB, p99	690 ms	11.6 ms
wrapped tool call, 10 KB, p50	4,067 ms	38 ms
wrapped tool call, 100 KB, p50	43,938 ms	355 ms

The target for the 1 KB wrapped call was p99 under 10 ms. It is close but not quite met on this laptop: p50 is 5 to 8 ms and p95 is under 10 ms, but p99 is 11 to 12 ms across runs (see "Noise" below).

How to reproduce¶

source .venv/bin/activate
python bench/make_dataset.py              # writes bench/data/detection.jsonl (seed 1337)
python bench/run_detection.py --label current
python bench/run_latency.py               # about 2 minutes now (was about 7)

Results land in bench/results/ as JSON plus a markdown table:

detection_baseline.*: detect.py before round 1
detection_after_extra_recognizers.*: after round 1 (this is "before" for round 2)
detection_after_fast_path.*: after round 2
latency_before_fast_path.*: timing run after round 1 (before round 2)
latency.*: timing run after round 2

Machine¶


CPU	12th Gen Intel Core i5-1235U (laptop chip, 12 logical CPUs)
RAM	15.3 GB
OS	Linux 7.0.0-38-generic
Python	3.12.3
Packages	presidio-analyzer 2.2.364, spacy 3.8.16, en-core-web-sm 3.8.0, opentelemetry-sdk 1.44.0, ff3 1.0.3

All timing is single-threaded. Laptop numbers are noisy (turbo, thermal), so treat them as "order of magnitude" figures, not exact ones.

Noise. The round 2 timing runs were taken while other jobs (load tests of the collector and gateway) were running on the same laptop, with a load average of 1 to 4. A re-run of the old code under that load was about 30% slower than its original run, so the "before" column uses the original, quieter run. The "after" numbers moved by up to 2x between runs at different load; the table uses a run started at a load average of 1.3. Treat a 1 KB p99 of "about 10 ms" as the honest figure.

The dataset¶

bench/data/detection.jsonl has 420 examples and 360 labeled entities, all synthetic and generated by a seeded script (running it twice gives the same file). Names are random pairs from common name lists, numbers are random digits in the right format, and checksums are valid where the format has one (Luhn for cards, mod-97 for IBANs).

group	examples	what is inside
positives	240	20 for each of 12 classes: US SSN, email, phone, credit card, IBAN, person name, street address, AWS access key, GitHub token, generic API key, JWT, private key
multi-entity records	24	CRM rows, emails and log lines with 2 to 4 entities each
hard negatives	108	12 each of: order IDs, UUIDs, timestamps, version numbers, git SHAs, zip codes, random 9-digit numbers, base64 blobs, hex hashes
evasion	48	12 each of: fullwidth digits, zero-width characters, spaced-out digits, value split across JSON fields

Every example is wrapped in one of five shapes an agent's tool I/O really has: JSON payload, chat message, log line, email, code or config file.

Caveat. There is no held-out set. The new rules in round 2 (street address pattern, SSN context words, key names for secrets) were written while looking at the false negatives on this same dataset. They are general rules, not lists of the dataset's values, but real data will score lower than this page shows. A second, independently written dataset would be the next step.

How it is scored¶

Span match: a detection counts if its character span overlaps a labeled span. "Strict" also needs the type to match (Presidio's LOCATION and the new STREET_ADDRESS count as an address). "Any type" accepts any overlapping detection, because for taint an SSN flagged as some other PII type still taints the chain.
Example level: "did detection flag anything in this example?" This is what decides taint in instrument_tool. It is measured on the raw text and in "payload" mode, where JSON examples are parsed and walked with detect_mapping the way the SDK does it.
Precision counts every sensitive detection that detect_text returns. Before round 2 that included DATE_TIME, ORGANIZATION and URL, because they tainted the chain. After round 2 they are not sensitive by default, so they are no longer returned (see below).

Detection results¶

Per class, strict span match, before and after round 2:

class	labels	precision before	precision after	recall before	recall after
US_SSN	43	1.00	1.00	0.65	0.81
EMAIL	47	1.00	1.00	1.00	1.00
PHONE	36	0.97	0.97	0.81	0.97
CREDIT_CARD	42	1.00	1.00	0.81	0.90
IBAN	20	1.00	1.00	1.00	1.00
PERSON	36	0.44	0.76	0.94	0.94
ADDRESS	28	0.87	0.96	0.68	1.00
AWS_ACCESS_KEY	26	1.00	1.00	0.85	0.85
GITHUB_TOKEN	22	1.00	1.00	0.77	1.00
GENERIC_API_KEY	20	0.46	1.00	0.80	0.95
JWT	20	1.00	1.00	1.00	1.00
PRIVATE_KEY	20	1.00	1.00	1.00	1.00
overall, strict	360	0.42	0.96	0.85	0.94
overall, any type	360	0.72	0.99	0.92	0.94

Example level ("was anything flagged?"):

mode	precision before	precision after	recall before	recall after	false alarms before	false alarms after
raw text	0.85	1.00	0.93	0.94	0.48	0.00
payload (`detect_mapping`)	0.84	1.00	0.94	0.94	0.52	0.00

All 108 hard negatives are now clean: none of the order IDs, UUIDs, timestamps, versions, SHAs, zip codes, random 9-digit numbers, base64 blobs or hex hashes is flagged.

Evasion cases (any-type hit rate):

evasion	before	after
fullwidth digits	12/12	12/12
zero-width characters	12/12	12/12
spaced-out digits	7/12	12/12
split across JSON fields	6/12 (noise hits only)	0/12

The split-field number went down, but the 6 "hits" before were all DATE_TIME or ORGANIZATION noise on a fragment, not real catches. This case is still not handled (see "Still weak").

What the recall change hides¶

Example-level recall stayed at 0.94 in payload mode, but what makes it up changed. 8 positives that used to be "caught" only through noise (a DATE_TIME hit on the integer ID next to the real value) are no longer flagged: 2 plain 9-digit SSNs with no SSN word nearby (member_ref, "Please verify ..."), and 6 split-field cases. In exchange 18 positives that were missed or only caught through noise are now caught for the right reason (plain SSNs next to ssn/"social security"/tax_id, github_pat_ tokens, sk_live_ keys, spaced digits, phones under a tel key, full street addresses). So the taint signal is much more trustworthy at the same recall.

Latency results¶

Warm numbers are after Presidio is built. The payload is a list of synthetic CRM rows (name, email, plan, note, sometimes an SSN or AWS key) whose JSON is about the given size. detect_text runs on one plain string of the same size. The wrapped tool is an echo tool under @instrument_tool(destination="external"), exporting to an in-memory span exporter, so there is no network time.

Round 2 adds a detection cache. To keep the numbers honest, the cache is cleared before every timed call, so every row is for a payload the SDK has never seen. The extra row "wrapped, same payload again" shows the cost when the exact payload was seen before (cache hit).

size	operation	p50 before	p50 after	p99 before	p99 after
100 B	detect_mapping	8.2	3.4	9.9	6.5
100 B	detect_text	5.1	5.6	10.7	9.2
100 B	redact_payload	5.6	2.6	7.0	4.1
100 B	wrapped tool	31.1	2.8	57.1	5.8
100 B	wrapped, same payload again		0.16		0.29
1 KB	detect_mapping	166	5.6	211	8.8
1 KB	detect_text	27	15	45	28
1 KB	redact_payload	101	30	122	48
1 KB	wrapped tool	518	8.4	690	11.6
1 KB	wrapped, same payload again		0.81		1.5
10 KB	detect_mapping	1,354	32	1,624	47
10 KB	detect_text	419	128	477	182
10 KB	redact_payload	949	164	1,234	209
10 KB	wrapped tool	4,067	38	4,174	61
10 KB	wrapped, same payload again		4.3		6.1
100 KB	detect_mapping	12,166	405	12,300	602
100 KB	detect_text	13,883	1,629	14,063	1,805
100 KB	redact_payload	10,579	396	11,859	453
100 KB	wrapped tool	43,938	355	47,162	432
100 KB	wrapped, same payload again		51		98

All times in ms. Cold init (median of 3 fresh processes) is about the same as before: 0.9 s to 1.7 s depending on load, most of it importing Presidio and loading the spaCy model.

Growth with size is now roughly linear for the wrapped call: about 10x time for 10x data from 1 KB up (8.4, 38, 355 ms). Before it was 518 ms, 4 s, 44 s.

Two things to read carefully:

redact_payload called on its own (outside a wrapped call) is still slower than detect_mapping, because it looks up each string separately and, with the cache cleared, each lookup is its own small detection pass. Inside a wrapped call it costs almost nothing, since detect_mapping has already found every value (see "per-call scope" below).
detect_text on one long block of prose is the slowest row. Its cost is spaCy's NER, which has to read every word of text full of capitalized names. That is linear but about 15 ms per KB of dense prose.

What changed in round 2¶

Speed¶

No more AnalyzerEngine.analyze per value. detect.py now runs Presidio's recognizers itself. Each pattern recognizer runs once over all values of a payload joined together (with a separator no regex can cross; a match that still crosses one is dropped and the recognizer re-runs on the values involved). Plain pattern recognizers are run with the same regexes and checksum validation, but without Presidio's per-call remove_duplicates, which was quadratic in the number of hits.
spaCy runs only the NER step, and only where it can matter. The tagger, parser and lemmatizer are switched off (they were only used by Presidio's context step). Values without a capitalized word (IDs, numbers, emails, lower-case notes) skip spaCy entirely. Short values are packed into shared documents (spaCy has a fixed cost per document) and long text is cut into 4,000-character windows with a 200-character overlap.
Cheap context words. Presidio's lemma-based context step was replaced with a regex look at the 5 words before a match, plus the value's JSON key. Same boost (+0.35) and same word lists as Presidio.
The phone matcher only sees values with 7+ digits. It was the single slowest recognizer. Two recognizers that can never reach the 0.5 threshold (US bank numbers at 0.05, passports at 0.1) are skipped.
Cache per value. A bounded LRU cache (4,096 entries, TAINTWARD_DETECT_CACHE_SIZE, 0 turns it off) maps a hash of each value to its detections. The cache holds only hashes and fingerprints, never the raw text.
Per-call scope. instrument_tool opens a detection_scope() around its input step and its output step. Inside it, every value is detected once by detect_mapping; the redactor's and the masker's own detect_text calls on the same strings reuse that answer. Before, each value was detected four times per call. taintward/sdk/redact.py did not need to change.

False alarms¶

DATE_TIME, ORGANIZATION, URL, NRP and MEDICAL_LICENSE are not sensitive by default. On this dataset they made up nearly all false alarms: DATE_TIME fired on integer IDs, zip codes and timestamps; ORGANIZATION on UUIDs and words like "SSN"; URL on the domain inside every email; MEDICAL_LICENSE on git SHAs (3 hits, 0 real). Turn any of them back on with TAINTWARD_PII_EXTRA_TYPES=DATE_TIME,URL or detect.set_pii_types(enable=[...]). Recognizers that only find ignored types are not run at all.
Person-name clean-up. A PERSON hit that contains digits or symbols (_ = @ / : and so on) or has no capitalized word is dropped; for log lines like full_name="Lena Chen" the name part is kept.
The generic API-key rule gives way to a JWT, GitHub, AWS or private key hit on the same characters, so one token is no longer reported twice.

Misses fixed¶

Plain 9-digit SSN when the JSON key or the words just before it say ssn, "social security" or tax_id (and the number passes SSN area/group rules). A 9-digit number with no such context is still never flagged, so tracking numbers stay clean. detect_mapping and instrument_tool now pass JSON keys and keyword-argument names along as context.
github_pat_... fine-grained tokens and ASIA... temporary AWS keys.
Generic keys: sk_live_ / sk_test_ / rk_ / pk_ prefixes, Slack xox?- tokens, Bearer <token>, and any credential-looking value (16+ characters with letters and digits) under a key such as api_key, client_secret, password or gh_token.
Spaced-out digits (7 2 4 - 6 8 - 6 9 6 6). The run is collapsed and only accepted if the result passes a real check: SSN rules, a Luhn checksum for cards, a valid number for phones. A plain list like "1 2 3 4 5 6 7 8 9" is not flagged.
US street addresses (<number> <Capitalized words> <St|Ave|Rd|...> with optional unit, city, state and ZIP), as a new STREET_ADDRESS type, so the whole address is masked and not only the city.

Also fixed on the way: the offset map for non-ASCII text now folds each character on its own, so a character that NFKC turns into two (a ligature) no longer shifts every later offset.

Tradeoffs¶

Not exactly Presidio any more. The context step is a cheap copy of Presidio's (same words, same boost, regex words instead of lemmas). On this dataset it found everything the old one did, but a word that only matches as a lemma ("calling" vs "call") can now be missed.
Names in lower case or in scripts without capitals are skipped by NER. The small English spaCy model rarely found those anyway, but it is a real gap for non-Latin names.
Dates, organizations and URLs no longer taint. That is the point, but a team that treats a date of birth or an internal URL as sensitive must turn the type back on. Turning DATE_TIME back on brings most of the false alarms back.
A value with SSN context under one key and none under another gets the stronger answer in both places inside one call (the scope keeps the union). This can over-redact, never under-redact.
The cache only helps repeated values. The headline numbers are with an empty cache. Agents often repeat the same values across calls (a customer record read and then written), and then the cost drops by about 10x.

Still weak¶

Values split across JSON fields (ssn_area/ssn_group/ssn_serial, a card in four parts, an AWS key in two halves): 0 of 12. Joining sibling fields whose keys share a prefix would catch most of them.
Plain 9-digit SSNs with no SSN word (member_ref: "547915995", "Please verify 259213629"): 3 of 7 still missed. Without context they look exactly like tracking numbers, and flagging all 9-digit numbers would bring back the false alarms.
Bare 32-character hex API keys without a key name: indistinguishable from hex hashes.
Person names: 2 names in JSON rows are missed by the small spaCy model, and capitalized words at the start of a line ("Cookie:", "Push") are sometimes called a person.
1 KB p99 is about 11 ms here, a bit over the 10 ms target. Most of the remaining time is spaCy NER on the name fields and the per-recognizer Python overhead.

Results on outside data¶

The synthetic dataset above was written together with the detection rules, so it flatters them. bench/external/ runs the same detector, scored the same way, on two public labeled PII datasets written by other people:

dataset	what it is	test sample
ai4privacy/open-pii-masking-500k-ai4privacy, validation split (CC-BY-4.0 + Llama 3.1/3.3 license)	short texts in en, fr, de, es, it, nl, hi, te	4,000 examples, 500 per language, 7,824 labeled values in a Taintward class
gretelai/synthetic_pii_finance_multilingual, test split (Apache-2.0)	full financial documents (emails, statements, XML, JSON) in 7 European languages	1,750 examples, 250 per language, 6,376 labeled values in a Taintward class

Both are also synthetic (written by LLMs), with their own label sets and formats.

How round 3 was kept honest. The first run on these datasets (before round 3) changed no rule. Round 3 then changed the rules while looking only at a dev sample: other rows of the same two datasets, never in the test sample (--split dev, 500 rows per language for ai4privacy, 140 for gretel). The numbers below are on the same test sample as the first run. The first run's list of misses did choose which areas to work on.

python bench/external/download.py                  # caches the files in ~/.cache/taintward-bench/external
python bench/external/run_external.py              # test sample, a few minutes
python bench/external/run_external.py --split dev  # dev sample, for tuning rules
python bench/external/run_external.py --check --wrapped 0   # what CI runs

Full results: bench/results/external_ai4privacy.md and bench/results/external_gretel.md. They show values only as shapes (Aa Aa, d3-d2-d4), never dataset text; tests/test_bench_external.py checks that.

What round 3 changed¶

Names in other languages. Text whose common function words are German, French, Spanish, Italian, Dutch or Swedish goes to spaCy's small multilingual model xx_ent_wiki_sm; English text and text with no such word (a lone name in a JSON field) stay with en_core_web_sm. Sending that second group to the multilingual model too was tried and dropped: it brought false alarms back on the English synthetic set (6.5% of clean payloads).
Fewer false name hits. Greeting and form label words ("Bitte", "Numéro"), template slots ([City]), words also written in lower case in the same text ("Policy" ... "policy"), spaCy stop words ("Unsere", "Estoy") and hits that start with a pronoun ("Meine Gedichte") are dropped. A stricter rule, keeping a one-word name only after a title or greeting, was tried and dropped: on the dev sample it removed more real first names (248) than false ones (201).
Phone numbers: a + country code with 9 to 15 digits, or a local number after a phone word in one of the seven languages.
Street addresses in the usual European forms ("Musterstraße 12", "12 rue de la Paix", "Via Roma 10", "Calle Mayor 5", "Kerkstraat 12a"), joined with a flat number and the postal code and city right after them, and more US street types from the USPS list.
National ids: with a check digit, anywhere (Aadhaar, codice fiscale, DNI / NIE, French NIR, Swiss AHV, UK National Insurance number, Indian PAN); any id-shaped value after an id word in the seven languages ("Passnummer", "numéro de passeport", "Codice fiscale") or under a key like passport_number. The plain word "id" is never enough.
Passwords and API keys written inline ("Passwort: ...", "API key: ...", <password>...</password>), when the value has a digit or symbol.
UK NHS numbers need their context word: the ten-digit check fired mostly on phone numbers.

Span recall per class, strict¶

class	synthetic	ai4privacy before	ai4privacy after	gretel before	gretel after
EMAIL	1.00	0.97	0.97	0.95	0.95
PERSON	0.94	0.50	0.54	0.48	0.48
PHONE	0.97	0.11	0.63	0.66	0.90
CREDIT_CARD	0.90	0.06	0.06	0.39	0.39
IBAN	1.00	-	-	0.68	0.68
SSN	0.81	0.06	0.09	0.29	0.69
ADDRESS	1.00	0.09	0.49	0.38	0.74
GOV_ID (non-US ids)	-	0.02	0.07	0.06	0.62
IP_ADDRESS	-	-	-	0.96	0.96
SECRET	0.95	-	-	0.18	0.45
overall recall	0.94	0.36	0.51	0.50	0.60
overall precision	0.96	0.57	0.78	0.30	0.43

Precision counts real PII that the dataset did not label as wrong (gretel labels no city or company), so it is a lower bound.

Example level: was anything sensitive found?¶

This is the number that matters most for taint, because one hit is enough to taint the call chain.

	synthetic	ai4privacy before	ai4privacy after	gretel before	gretel after
recall	0.94	0.68	0.75	0.94	0.94
precision	1.00	0.97	0.98	0.87	0.88
false alarm rate on examples with nothing to find	0.00	0.26	0.15	0.66	0.62

Per language on ai4privacy, the false alarm rate dropped most where the English model was worst: German 0.85 to 0.28, Italian 0.39 to 0.25, Dutch 0.22 to 0.09. Strict recall rose in every language, most in Italian (0.38 to 0.66), Spanish (0.30 to 0.55) and French (0.39 to 0.61).

Still weak on outside data¶

False alarms on long documents (0.62 of gretel documents with nothing labeled). Most are real place and company names the dataset does not label ("Berlin", "Stockholm", "ABCDEF GmbH") and title-case headings. A place name on its own counts as sensitive today (LOCATION); treating a lone city or country as not sensitive would remove a large part of these, but it is a product decision, not a detection fix.
Names: about half are found. The small models miss many one-word first names in running text, and names in non-Latin scripts (hi, te) or in lower case are not looked for at all.
Ids with no id word and no check digit: most ai4privacy ids are random strings with nothing before them ("please send your RQ7118854"), so GOV_ID is 0.07 and SSN 0.09 there.
Cards and IBANs that fail their checksum are still rejected on purpose (87 of 101 cards in ai4privacy fail it).
Address parts on their own: a lone postal code or house number.

Latency on outside data¶

	before	after
`detect_text`, short texts (<200 chars), p50 / p99	2.8 / 4.9 ms	2.1 / 4.2 ms
`detect_text`, gretel documents (1k-5k chars), p50 / p99	16.1 / 27.3 ms	15.0 / 37.6 ms
wrapped tool call, ai4privacy, p50 / p99	5.8 / 10.4 ms	4.3 / 8.3 ms
wrapped tool call, gretel, p50 / p99	34.2 / 60.9 ms	27.4 / 61.0 ms

The "after" run had a load average of about 1.5. Run back to back on the same 600 documents of 1k-5k characters, twice each, the old code took 15 to 20 ms (p50) and 23 to 40 ms (p99), the new code 16 ms and 30 to 31 ms: the multilingual model is cheaper than the English one on text in other languages, and the new rules add about 1 ms per document. On short texts both took 2.6 to 3.4 ms.

Regression guard¶

The CI job detection-bench (GitHub and GitLab) downloads the two outside datasets at their pinned commit, runs the test sample with run_external.py --check and fails when a score drops under its floor (FLOORS: overall strict recall and precision, example recall, false alarm rate, and recall of the classes round 3 improved), or when the multilingual model is not loaded. The floors sit 0.01 to 0.03 under the numbers above.
tests/test_detect_intl.py covers every round 3 rule, with cases that must be found and cases that must be left alone. Each rule was also checked by putting its bug back: 44 of 47 such changes make a test fail; the other 3 remove a quick pre-check whose condition the inner check repeats, or code that no longer exists.
tests/test_bench_smoke.py runs detection on 25 fixed positive examples and 7 hard negatives. It fails if recall on US SSN (including plain and spaced ones), AWS key, email, JWT, private key, GitHub token, address or credit card drops below 1.00 on those examples, or if any hard negative gets flagged at all. When bench/data/detection.jsonl is present it also runs the full benchmark and fails if payload recall drops below 0.94, payload precision below 1.00, strict span precision below 0.96, strict span recall below 0.94, or the false alarm rate goes above 0.00.
tests/test_detect_latency.py times 15 wrapped calls on the 1 KB payload with the cache cleared and fails if the median is over 50 ms (5 to 10x what this laptop measures, so a slow CI runner still passes). Set TAINTWARD_SKIP_LATENCY_TEST=1 to skip it, or TAINTWARD_LATENCY_BOUND_MS to change the bound.
tests/test_detect_fast.py covers each new rule, the ignored types, the cache, the per-call scope and offset mapping.