Python API¶
Generated from the docstrings in the source. Only the names listed on Stability and versioning are public API; everything else in the package can change in any release.
import taintward loads nothing heavy. The names below are loaded on first
use, so import taintward.core.doe does not load Presidio or spaCy.
| name | where it lives |
|---|---|
tw.init, tw.run, tw.restore, tw.get_ward, tw.Ward |
taintward.sdk.integrations |
tw.wrap_langchain_tools, tw.protect_langchain_model |
taintward.sdk.integrations (needs the langchain extra) |
tw.wrap_function_tools |
taintward.sdk.integrations (needs the openai-agents extra) |
tw.protect_openai, tw.protect_anthropic |
taintward.sdk.integrations (needs the llm extra or the provider SDK) |
tw.health |
taintward.sdk.detect |
Integrations¶
Low-code Taintward integration for agent frameworks.
Instead of putting @instrument_tool(...) on every tool by hand, call
init() once with a tool-name -> destination map, then hand your framework
objects to an adapter::
from taintward.sdk import integrations as ward
ward.init("hr-agent", destinations={
"search_db": "internal",
"send_email": ("external", "us"), # (destination, jurisdiction)
})
tools = ward.wrap_langchain_tools(tools) # LangChain / LangGraph
model = ward.protect_langchain_model(model) # LLM requests are redacted
client = ward.protect_openai(OpenAI()) # plain OpenAI client
agent_tools = ward.wrap_function_tools(tools) # OpenAI Agents SDK
The same pair (tools + model) exists for CrewAI (wrap_crewai_tools,
protect_crewai_llm), LlamaIndex (wrap_llamaindex_tools,
protect_llamaindex_llm), Pydantic AI (wrap_pydantic_ai_tools,
protect_pydantic_ai_model) and Google ADK (wrap_adk_tools,
protect_adk_model). protect_litellm() covers litellm.completion;
the LiteLLM proxy uses taintward.sdk.integrations.litellm.TaintwardGuardrail.
Unmapped tools default to external. That is the safe choice: an unknown
tool may send data anywhere, so it gets the non-reversible mask and its span
is flagged when a tainted chain reaches it. Marking a tool internal is an
explicit, reviewable decision to trust it with reversible tokens. A tool that
was wrongly left external only loses some usefulness (it sees [PII]); a
tool wrongly treated as internal would receive data it should not.
Placeholders (on by default): the LLM does not get [PII] masks, it gets
stable typed placeholders such as <PERSON_1> and <SSN_1>, so it can
still do the task. The model's answer and tool calls are restored to the real
values before they reach your code, and each tool then gets what its
destination allows (see taintward.sdk.placeholders and docs/integrations.md). Turn
it off with init(..., placeholders=False) to go back to masks.
All redaction and detection is done by the existing SDK (taintward.sdk.detect,
taintward.sdk.redact.redact_payload, taintward.sdk.instrumentation.GuardedCall). The
adapters only translate each framework's call shape into that one gate.
Ward ¶
Holds the tool map, the LLM policy and the tracer used by adapters.
with_overrides ¶
A copy with extra/overriding tool entries (used by wrap_*(mapping=)).
run ¶
One agent run: one trace, a clean taint slate, taint that follows the run across framework threads and asyncio tasks, and one placeholder map shared by every LLM call and tool in the run.
conversation_id with a placeholder_vault keeps the
placeholders of a long-lived conversation in the vault, so later runs
of the same conversation reuse them. Without a vault the map lives in
memory and is cleared when the run ends.
placeholder_map ¶
The run's placeholder map for tool calls (None when placeholders are off or no run is active).
llm_placeholder_map ¶
The map for one LLM request: the run's map, or a fresh map for this one call when no run is active (never shared between callers).
restore ¶
Swap placeholders back to real values with the current run's map
(for output the adapters do not restore, e.g. collected stream text).
Each call is recorded with redact.record_reversal (gap G11).
ToolPolicy
dataclass
¶
Where one tool sends its input.
init ¶
init(service_name: str, endpoint: str | None = None, destinations: Mapping[str, Any] | None = None, *, default_destination: str = DEFAULT_DESTINATION, default_jurisdiction: str = '', llm_jurisdiction: str = '', console: bool = False, tracer: Tracer | None = None, placeholders: bool = True, placeholder_vault: Any = None, placeholder_ttl: float | None = None, placeholder_instruction: bool = True, placeholder_strict: bool = False, placeholder_restore_in_urls: bool | None = None) -> Ward
Configure Taintward once per process and return the global Ward.
service_name/endpoint/consolego toinstrumentation.configure_tracing(OTLP/HTTP to SigNoz).destinationsmaps tool names to a destination, e.g.{"send_email": "external", "search_db": "internal"}. A value can also be("external", "eu")or{"destination": .., "jurisdiction": ..}.default_destinationis used for tools not in the map (defaultexternal, the safe choice).llm_jurisdictiontags LLM request spans (e.g."us").tracer: pass your own tracer to skip global tracing setup (tests, or an app that already configured OpenTelemetry).placeholders(default True): the LLM gets stable placeholders such as<PERSON_1>instead of[PII], and the model's output is restored. False goes back to plain masks.placeholder_vault/placeholder_ttl: optionaltaintward.sdk.vaultbackend (and entry lifetime) for placeholders of long-lived conversations, used byrun(conversation_id=...).placeholder_instruction(default True): add one short system line asking the model to copy placeholders exactly. False sends the request as it is.placeholder_strict(default False): raiseplaceholders.PlaceholderRestoreErrorwhen the model's answer has placeholders that could not be restored, instead of a warning.placeholder_restore_in_urls(default off, orTAINTWARD_RESTORE_IN_URLS=1): also restore placeholders inside link and image addresses. Off, an injectedkeeps the placeholder instead of carrying the real value out (gap G1).
run ¶
Context manager for one agent run on the global Ward.
The framework modules (taintward.sdk.integrations.crewai, .llamaindex,
.pydanticai, .google_adk, .litellm) import their framework, so they
are described in SDK integrations instead of here.
Decorator and low-level tracing¶
OTel instrumentation for agent tool calls: the runtime taint tracker.
Wraps each tool call in an OTel span using GenAI semantic conventions
(gen_ai.tool.*) and propagates the taint label via OTel baggage. This is
AgentRaft's LA-DTP (runtime taint tracking) on OpenTelemetry, with Φ replaced
by structural baggage propagation plus per-I/O detection re-fire (see
docs/agentraft-mapping.md sections D/E).
Per tool call the wrapper:
- reads the incoming chain-level taint from baggage,
- detects sensitive data in the tool's input (field-level taint),
- merges input taint with the chain taint and attaches it to the context so nested calls inherit it (this is the value-independent propagation that survives any LLM transformation),
- runs the tool,
- detects sensitive data in the tool's output and merges again,
- materializes the merged taint as span attributes,
- stores the tool I/O on the span masked so SigNoz never stores raw PII,
- tags the span with its
destination(internal/external/llm/log/rag) and, for external/llm sinks, records an SDK-side egress violation when the chain is tainted (the Phase 4 redaction gate hardens this from "record" to "block").
Span links model fan-out/fan-in (one parent, many children, cross-trace), used by the RAG tool's retrieval fan-out in the demo. Parent-child links for linear flows are automatic via OTel context.
GuardedCall ¶
One call into a sink, run under the egress gate.
This is the shared core behind instrument_tool and the framework
adapters in taintward.sdk.integrations. Use it as a context manager::
with GuardedCall("send_email", destination=DEST_EXTERNAL,
kwargs={"to": to, "body": body}) as call:
result = real_send(*call.args, **call.kwargs) # redacted args
call.record_result(result)
On enter it reads the chain taint, detects sensitive data in the input,
redacts the input for destination and opens the span. call.args /
call.kwargs hold the redacted input the sink must receive. On exit it
sets the egress violation flag, ends the span and carries the outgoing
taint forward to the next call. It works the same inside async code.
redact replaces redact_payload when a caller needs a structure-aware
walk (LLM message lists). passthrough names keyword args that are an
explicit, reviewed exception and reach the sink unredacted (they are still
scanned, so the taint label stays correct). attributes adds extra span attributes and
input_attr / output_attr name the attributes that store the
(redacted / masked) input and output.
record_result ¶
Detect taint in the sink's output and store it masked on the span.
view is what gets scanned and stored when the raw result is not a
plain value (for example a framework message object).
configure_tracing ¶
configure_tracing(service_name: str, *, endpoint: str = DEFAULT_OTLP_HTTP, console: bool = False) -> trace.Tracer
Set up the global TracerProvider with an OTLP/HTTP exporter to SigNoz.
Call once at process start. Returns the tracer to pass to instrument_tool
(or use trace.get_tracer(...) elsewhere). console=True adds a
console exporter for debugging without SigNoz.
instrument_tool ¶
instrument_tool(name: str | None = None, *, destination: str = DEST_INTERNAL, jurisdiction: str = '', tracer: Tracer | None = None)
Decorator that wraps a tool function with taint-aware OTel tracing.
The wrapped function's positional/keyword args are treated as the tool's
input payload; the return value as the output. The caller may pass an
extra taintward_links=[span_context, ...] kwarg to add span links
for fan-out/fan-in (consumed, not forwarded to the tool). async def
tools are supported: the span then covers the awaited call.
begin_run ¶
Start a run: attach a root span as current so every tool span in the
run shares one trace, and reset the taint slate. Pair with end_run().
tracer defaults to the global taintward tracer.
Runs nest: end_run ends the innermost run and the outer one goes on.
end_run ¶
End the innermost run: restore the context from before its
begin_run (clearing its taint) and end its root span. With no run
active it clears the taint left by calls made outside a run.
current_taint ¶
Convenience: the taint label on the current OTel context (including taint recorded earlier in the current run).
Detection¶
Sensitive-data detection for the Taintward SDK.
Two detection families:
- PII via
presidio-analyzerrecognizers (regex + checksums) and spaCy NER: SSN, email, phone, credit card, person names, places, etc. - Secrets via custom recognizers: AWS access keys, GitHub tokens, JWTs, private keys and a generic API-key heuristic, so the demo breach-simulator can inject a credential, not just an SSN.
Output is a :class:Detection list carrying only sensitivity classes and a
short non-reversible fingerprint of the matched value, never the value
itself. The fingerprint lets the lineage map correlate the same secret across
spans without re-storing it.
How it stays fast (numbers in docs/benchmark.md):
- Presidio's
AnalyzerEngine.analyzeis not called per value. The pattern recognizers run directly, once over all the values of a payload joined together, and spaCy runs with only its NER step, only on values that have a capitalized word in them (IDs, numbers and lower-case notes skip it). - Context words ("ssn", "phone", ...) are looked up with a cheap regex in a short window before each match and in the JSON key of the value, instead of Presidio's lemma-based context step, which is slow on long text.
- Results are cached per value (keyed by a hash, so the cache never holds the
raw text), and inside one tool call (
detection_scope) every later lookup of the same value reuses the first result.
Entity types that fire on almost everything (DATE_TIME, ORGANIZATION,
URL, NRP, MEDICAL_LICENSE) are not treated as sensitive by
default. TAINTWARD_PII_EXTRA_TYPES=DATE_TIME,URL (or
:func:set_pii_types) turns any of them back on.
Values longer than TAINTWARD_DETECT_MAX_CHARS (default 500000
characters; 0 turns the limit off) are not scanned: the whole value is
treated as one secret (entity type OVERSIZE_VALUE), so it is masked
instead of holding the call for seconds (gap G10).
DetectionDegradedError ¶
Bases: RuntimeError
Raised (with TAINTWARD_FAIL_CLOSED=1) when a call to a non-internal
destination is attempted while detection runs in a degraded mode.
Detection ¶
A single detected sensitive span.
health ¶
State of the detector: {"ok": bool, "mode": str, "error": str|None,
"multilingual_model": str|None}.
mode is "full" (Presidio + spaCy NER), "regex_only" (the NER
model or Presidio failed to load; patterns, checksums and key context
still work), "broken" (detection itself raised) or "not_loaded"
(load=False before the first detection). multilingual_model names
the model used for names in text that is not English (xx_ent_wiki_sm
when it is installed), or is None. With load=True (default) the
detector is loaded first, so the answer is real.
set_pii_types ¶
Choose which noisy entity types count as PII.
enable turns types such as DATE_TIME or URL back on (they then
taint like any other PII); disable stops a type from counting. Returns
the resulting set of ignored types. Clears the detection cache.
detect_text ¶
Detect sensitive spans in text. Returns [] if none / on error.
Presidio is loaded lazily. If the spaCy model or Presidio cannot be
loaded, detection falls back to regex-only mode (see :func:health)
instead of returning nothing. An ERROR is logged once.
detect_mapping ¶
Detect across a JSON-like payload (dict / list / scalar).
Redaction¶
Destination-aware redaction: the Phase 4 standout feature.
The real "context-aware" redaction: the same sensitive value is treated differently depending on where it is going, because Phase 2/3 taint tracking followed where the data actually travels (not a static field-name mask).
internal sink -> format-preserving REVERSIBLE token
(the internal store can still use/correlate a value
of the same shape/length without holding raw PII;
the token reverses back to the original)
external / llm / log -> non-reversible [PII] / [SECRET] mask
(the value must not leave, and cannot be recovered
from what is stored)
rag -> mask (embeddings are invertible, OWASP LLM08:2025)
Placeholder mode (placeholders= a taintward.sdk.placeholders.PlaceholderMap):
the llm destination gets a typed, numbered placeholder such as
<PERSON_1> instead of the mask, stable within the map, so the model can
still tell values apart. Every other destination is unchanged, but values the
map already knows are redacted even where the detector misses them.
Implementation: FF1 format-preserving encryption (NIST SP 800-38G, see taintward/sdk/ff1.py) makes the token (same length, digits stay digits, separators kept in place). A pluggable vault (taintward/sdk/vault.py) maps token -> exact original so reversal is exact even where FPE alone cannot be (letter case is not kept in a radix-36 token). Values too short for FF1's minimum domain size (radix**len >= 1,000,000, so fewer than 6 digits or 4 letters/digits) are masked instead of tokenized.
Keys (details in docs/redaction-crypto.md and docs/key-management.md): TAINTWARD_FPE_KEY hex AES key (32/48/64 hex chars). Keep it secret. TAINTWARD_FPE_KEY_ID optional label stored with each token in the vault TAINTWARD_KEYS key ring source (file, HashiCorp Vault, KMS wrapped keys); its active FPE key is used when TAINTWARD_FPE_KEY is unset TAINTWARD_ALLOW_DEMO_KEY =1 to use the built-in demo key (demos only) If no key is set and the demo key is not allowed, a random key is generated once per process and a warning is logged: tokens then differ after a restart.
Redactor ¶
Destination-aware redactor: reversible FPE token for internal, mask elsewhere.
vault: where token -> original is kept. Default comes from TAINTWARD_VAULT (MemoryVault when unset). structural_fallback: when the vault has no entry, try to decrypt a digit-only token with the key. Default: on for MemoryVault (tokens from another process can still be reversed), off for persistent vaults, so that deleting a vault entry (GDPR erasure) really makes the token unreversible through this API. ttl: optional lifetime in seconds for vault entries.
redact_value ¶
redact_value(value: str, sensitivity: Sensitivity, destination: str, *, subject: str | None = None) -> str
Redact one value for a destination.
internal -> reversible FPE token (vaulted for exact reversal; falls
back to a mask if FPE can't handle the value, e.g. too short); any
other sink -> non-reversible mask. subject (optional, e.g. a user
id) tags the vault entry so forget_subject() can erase it later.
redact_text ¶
redact_text(text: str, detections, destination: str, *, subject: str | None = None, placeholders=None) -> str
Apply redaction to each detected span in text (right-to-left).
Overlapping detections (e.g. AWS_ACCESS_KEY and GENERIC_API_KEY on the same region) are deduped to the longest per region so the output is well-formed regardless of how many recognizers match. For spans of the same length a SECRET wins over PII, then the higher score.
placeholders (a PlaceholderMap): for the llm destination each
span becomes its stable placeholder (<SSN_1>) instead of a mask.
reverse ¶
Reverse an internal-sink FPE token to the exact original.
Returns None for masks or unknown tokens. Masks are non-reversible by
construction, that's the whole point of the external/llm/log path.
Every call is recorded with record_reversal (gap G11).
forget_subject ¶
GDPR erasure: delete every vault mapping tagged with this subject.
With structural_fallback on (the MemoryVault default), digit-only tokens can still be decrypted by anyone holding the FPE key, so erasure is only complete with structural_fallback off (the default for persistent vaults).
redact_payload ¶
redact_payload(value, destination: str, redactor: 'Redactor | None' = None, *, subject: str | None = None, placeholders=None)
Recursively redact sensitive values in a structured payload for the given destination. Non-sensitive values and structure are preserved. Used by the SDK egress gate on tool input args.
- Containers: dict (and dict subclasses such as OrderedDict), list, tuple, namedtuple, set and frozenset keep their type; other Mapping / Sequence types become a dict / list.
- Dict keys are redacted like values. When two redacted keys come out the
same, a numeric suffix keeps them apart (
[PII],[PII]_2). - Leaves that are not strings (int, float, Decimal, any object) are
checked in their
strform; a sensitive one is replaced by the redacted string (a mask, or a token forinternal). None and booleans are kept. - Outside a
detection_scopethe whole payload is detected first in one pass, so JSON keys count as context ({"ssn": 234121234}).
placeholders (a taintward.sdk.placeholders.PlaceholderMap, optional): the
llm destination gets stable placeholders instead of masks, and for
every destination the values the map already knows are redacted even
where the detector does not find them.
Token vault¶
Token vault: token -> exact original value, for reversing internal tokens.
FPE alone cannot give back the exact original in every case (taintward/sdk/redact.py lowercases letters before encrypting, and short values are masked), so the Redactor keeps a mapping from each token it issues to the original value. This module makes that mapping pluggable:
MemoryVault in-process dict (the old behaviour). Lost on restart and
not shared between processes. Default.
SQLiteVault persistent file, safe for several processes on one host
(WAL mode + busy timeout). The original value is stored
ENCRYPTED with AES-256-GCM under a vault key that is
separate from the FPE key. Plaintext is never written.
PostgresVault shared table in PostgreSQL (psycopg 3), for several
replicas on several hosts. Same encryption as SQLiteVault.
RedisVault shared Redis (redis-py), native TTL, one set per subject
blind index for erasure. Same encryption as SQLiteVault.
Every backend supports: put(token, original, ttl=None, subject=None, key_id=None) get(token) -> original or None (expired rows count as missing) delete(token) -> bool forget_subject(subject) -> number of tokens erased (GDPR erasure)
Subjects (e.g. a user id or email) are stored only as an HMAC-SHA256 blind index, so the vault file does not list who the data belongs to.
Configuration (see get_vault()):
TAINTWARD_VAULT "memory" (default), "sqlite://
psycopg and redis are optional dependencies: they are imported only when a PostgresVault or RedisVault is created.
Vault ¶
Bases: Protocol
MemoryVault ¶
In-process vault. Fast, but lost on restart and not shared.
SQLiteVault ¶
Persistent vault in a SQLite file, original values encrypted at rest.
Each row stores: token, a random 12-byte nonce, AES-GCM ciphertext of the original (the token is the associated data, so a ciphertext cannot be moved to another token's row), an optional expiry time, an optional HMAC blind index of the subject, and the FPE key id used.
Safe for multiple processes on the same host: WAL journal mode, a busy timeout, and one short transaction per call. Do not put the file on a network filesystem (SQLite locking is not reliable there).
reseal ¶
Re-encrypt every row not sealed with the active vault key.
Run it after rotating the vault key; then the old key can be retired. The subject index is rebuilt under the active key too. Rows written before key rings do not hold their subject, so their index is kept; erasure finds them while the key that wrote them is in the ring.
PostgresVault ¶
Vault in a PostgreSQL table, shared by any number of replicas.
One row per token: the token (primary key), a 12-byte nonce, the AES-GCM ciphertext of the original (token as associated data), the FPE key id, the HMAC blind index of the subject and the expiry time. Plaintext and subject ids are never sent to the server.
Expiry uses the database clock (now()), so replicas with skewed
clocks still agree on what has expired. put is a single
INSERT ... ON CONFLICT (token) DO UPDATE, so concurrent writers never
fail on a duplicate key. There is an index on the subject index (for
erasure) and a partial index on the expiry (for purge_expired).
Connections come from a small built in pool (pool_size), each in
autocommit mode with a connect timeout and a statement timeout. A broken
connection is dropped and the call is retried once on a fresh one. After
os.fork() the child starts with an empty pool.
reseal ¶
Re-encrypt every row not sealed with the active vault key (see SQLiteVault.reseal). An update only lands if the row was not rewritten in the meantime.
RedisVault ¶
Vault in Redis, shared by any number of replicas.
Each token is a hash <prefix>t:<token> holding the sealed original
(nonce || AES-GCM ciphertext, token as associated data), the FPE key
id and the subject blind index. A TTL is a native Redis expiry, so
expired tokens disappear without a purge job. Each subject blind index
has a set <prefix>s:<index> of its tokens, used by
forget_subject. Writes and erasure run as Lua scripts, so each one is
atomic.
redis-py keeps a connection pool (max_connections), is safe to use
from several threads, and opens new connections after a fork. The
erasure script reads keys it builds itself, so use a single Redis (a
primary with replicas or Sentinel is fine), not Redis Cluster.
Run Redis with persistence (appendonly yes): a Redis that loses its
data loses the token mapping.
reseal ¶
Re-encrypt every token not sealed with the active vault key (see SQLiteVault.reseal). HSET keeps the key's TTL; the swap only lands if the value did not change since it was read.
VaultError ¶
Bases: RuntimeError
The vault backend could not be reached or used (connection refused, timeout, missing driver). The message never contains the password.
get_vault ¶
Build a vault from a spec string or TAINTWARD_VAULT.
"memory" (or empty) -> MemoryVault
"sqlite://
Key rings¶
Key providers and rotation for the FPE key and the vault key.
Taintward uses two secret keys:
fpe AES key for FF1, makes the reversible tokens (taintward/sdk/redact.py)
vault AES-GCM key that seals originals in the vault (taintward/sdk/vault.py)
Each purpose has a key ring: one active key that is used for new work, and older keys that are only used to read what they made. Where the rings come from is set with TAINTWARD_KEYS:
unset or "env" TAINTWARD_FPE_KEY / TAINTWARD_VAULT_KEY (one key
each), plus TAINTWARD_VAULT_KEY_PREVIOUS (comma
separated hex) for vault keys being rotated out
"file:<path>" a JSON key ring file (format below)
"vault-kv:<mount>/<path>" the same JSON stored in HashiCorp Vault KV v2,
read with VAULT_ADDR and VAULT_TOKEN
TAINTWARD_FPE_KEY and TAINTWARD_VAULT_KEY, when set, still win over the ring for the active key, so an existing setup keeps working unchanged.
Key ring JSON:
{
"kms": "aws-kms:alias/taintward", optional default for wrapped keys
"fpe": {"active": "2026-10",
"keys": {"2026-10": {"wrapped": "<base64>"},
"2026-04": "<hex>"}},
"vault": {"active": "v2", "keys": {"v2": {"hex": "<hex>"}}}
}
A key is plain hex (a string or {"hex": ...}) or wrapped by a KMS ({"wrapped": ..., "kms": ...}, "kms" falling back to the top level one). Wrapped keys are unwrapped once, when the ring is loaded:
aws-kms:<key id, arn or alias> AWS KMS Decrypt (boto3, AWS_* settings)
gcp-kms:projects/.../cryptoKeys/<k> Google Cloud KMS (google-cloud-kms)
vault-transit:[<mount>/]<key> HashiCorp Vault Transit decrypt
AWS and GCP wraps are bound to the purpose ("taintward-key" = "fpe" or "vault" as encryption context / additional data), so a wrapped vault key cannot be passed off as an FPE key.
The taintward-keys command makes and rotates rings: generate, rotate, retire, show and reseal (re-encrypt vault rows under the active key). See docs/key-management.md.
KeySet
dataclass
¶
KeyRing
dataclass
¶
The keys of one purpose: active makes new work, the rest only read.
KeyConfigError ¶
Bases: ValueError
A key ring could not be loaded or is invalid. Never holds key bytes.
load_keys ¶
Load the key rings named by spec or TAINTWARD_KEYS.
A ring from a file or Vault is loaded (and unwrapped) once per process and cached; call reload_keys() after rotating.
wrap ¶
Wrap key bytes with the named KMS; the result goes in a key ring.
unwrap ¶
Turn a wrapped key back into key bytes with the named KMS.
Placeholders¶
Consistent placeholders for the LLM, and restore on the way back.
The problem with masks: when the model gets [PII] it cannot do its job.
"Write to [PII] about her tax form" has lost who the letter is for, and two
different people both become [PII].
Placeholders fix that. Every sensitive value sent to the LLM is replaced by a typed, numbered, stable placeholder::
"Alice Lee" -> <PERSON_1>
"234-12-1234" -> <SSN_1>
"AKIAIOSFODNN7EXAMPLE" -> <AWS_KEY_1>
"bob@example.com" -> <EMAIL_1>
Within one run (one conversation) the same value always gets the same placeholder and different values get different numbers, so the model can still reason about "who is who". The model writes its answer and its tool calls with the placeholders, and Taintward swaps them back:
model answer -> caller (end user) real values
model tool call -> any tool real values first, then the tool's
normal destination gate runs
(internal: reversible token,
external/log/rag: mask, unless the
argument is in the tool's allow list)
The map (placeholder -> real value) is kept in memory for the run only. It is
never sent anywhere and it is cleared when the run ends. For long-lived
conversations it can be backed by a token vault (taintward.sdk.vault), keyed by a
conversation id; the vault entries are tagged with a subject so one call to
forget_conversation() erases them.
PlaceholderMap.find_known also lets the redactor redact a value the map
already knows even when the detector misses it in a new context (for example
a bare name in a tool argument), so restoring never lets a known value slip
past the gate.
Placeholders inside a link or image address are not restored (gap G1 in
docs/threat-model.md): an injected 
would otherwise become an image request that carries the real SSN as soon
as the answer is rendered as markdown or HTML. They stay as placeholders and
are reported as blocked. restore_in_urls=True (or
TAINTWARD_RESTORE_IN_URLS=1) turns the check off.
PlaceholderMap ¶
Placeholder <-> real value map for one run (one conversation).
vault: optional taintward.sdk.vault backend for long-lived conversations. Only
the public vault API is used (put/get/delete/forget_subject).
conversation_id: required with vault; names the conversation whose
placeholders are stored. It is stored only as a keyed hash.
ttl: lifetime in seconds of vault entries.
index_key: secret key for the keyed hashes used in vault keys. Default is
derived from the FPE key (taintward.sdk.redact.get_redactor().key), so vault
keys do not reveal values or conversation ids.
restore_in_urls: also restore placeholders inside link and image
addresses. Default False (TAINTWARD_RESTORE_IN_URLS), see gap G1.
Thread safe: frameworks run tools in threads that share the run's map.
placeholder_for ¶
placeholder_for(value: str, entity_type: str | None = None, sensitivity: Sensitivity = Sensitivity.PII) -> str
The stable placeholder for value; a new one if it is unseen.
lookup ¶
Real value for a placeholder (any spacing/case variant), or None.
find_known ¶
Spans of values this map already knows, found by exact match.
Used together with the detector so a known value is redacted even where the detector would miss it (for example a bare name).
restore_text ¶
Swap every known placeholder in text for its real value.
Unknown placeholders (for example ones the model invented) are left as
they are, and so are placeholders inside a link or image address
(unless restore_in_urls). A string that is a JSON object or array
(tool call arguments) is restored inside the parsed JSON, so a real
value with a quote or backslash stays valid JSON.
before: text that came before text (earlier stream chunks),
used only to see whether a placeholder continues a link address.
spaced=False: do not restore "Person 1" style forms (a streamed
answer, where the rest of the text is not known yet).
restore_json_fragment ¶
Like restore_text, for text that sits inside a JSON document
that is not complete yet (a streamed piece of tool call arguments):
the real value is JSON-escaped, so the arguments stay valid JSON.
diagnose ¶
What restoring text (a model answer, before restore) will do.
spaced=False for a streamed answer: stream pieces never restore
"Person 1" style forms, so they count as lost.
Returns lists of placeholder forms (never values):
repaired: rewritten forms that tolerant restore fixes, for example[PERSON_1]orPerson 1.lost: forms that point at a known placeholder but are not safe to restore (<PERSON>with no number,person_1in lower case,Person 1next to a real<PERSON_2>). The caller gets this text as the model wrote it.unknown: placeholders that are not in the map, for example an invented<PERSON_3>.blocked: known placeholders inside a link or image address, left as placeholders on purpose (gap G1).
restore ¶
Restore placeholders in a str / dict / list / tuple (a new object).
restore_inplace ¶
Restore placeholders inside a provider response object.
Dicts and lists are changed in place, pydantic models (OpenAI and Anthropic SDK responses) get their fields set. Returns the object, or a new value for str/tuple.
forget_conversation ¶
Erase this conversation's placeholders from memory and the vault.
restore ¶
Restore placeholders with pmap (default: the current run's map).
Use it for output the adapters do not restore themselves, for example text you collected from a stream.
Evidence log¶
Sharded evidence log: one Merkle log file per tenant per UTC day.
Layout::
<dir>/<tenant>/<YYYY-MM-DD>.logl one MerkleLog per tenant per day
<dir>/_checkpoints/<seq>.json signed checkpoints (taintward/evidence/checkpoint.py)
Why shard: a single file grows forever, has one writer, and must be read in full to compute the root. Per-day shards stay small, can be archived or moved to object storage once the day is over, and a verifier can check one shard without the others. Signed checkpoints tie all shards together, so a deleted or rewritten shard is still detected.
Each shard is a normal :class:taintward.evidence.merkle.MerkleLog file, so every
existing tool (taintward-evidence --log-file <shard> root) works on it.
Writers in several processes are safe: each append takes an exclusive lock
on the shard file, reloads the shard if another process changed it since the
last read, and fsyncs the new line (see :class:taintward.evidence.merkle.MerkleLog).
ShardedLog ¶
Merkle evidence log split by tenant and UTC day.
checkpoint_every > 0 with a signing_key writes a signed
checkpoint automatically after that many appends from this process.
shard ¶
The shard's MerkleLog, reloaded if the file changed on disk.
find ¶
Locate a record by trace and span id: (tenant, day, index).
append ¶
append(record: dict, *, tenant: str = DEFAULT_TENANT, day: str | None = None, unique: bool = False) -> tuple[str, str, int] | None
Append record to its shard; return (tenant, day, index).
The day defaults to the UTC day of record["timestamp_ns"] (or
today). With unique=True a record whose (trace_id, span_id) is
already in the shard is skipped and None is returned, so recording
the same time window twice does not double count.
checkpoint ¶
Write a signed checkpoint over every shard now on disk.
repair ¶
Move a broken last line (left by a crash) out of every shard.
See :func:taintward.evidence.merkle.repair_log.
Tamper-evident Merkle log of Taintward violations.
Each violation the collector's Rego policy flags is appended as a leaf. The log is a Merkle hash tree over SHA-256 in the shape defined by RFC 6962 / RFC 9162 (Certificate Transparency), stored as an append-only file with one canonical JSON record per line. Anyone can prove that a record is in the log (inclusion proof) and that a newer log extends an older one without changing it (consistency proof). Both proofs are O(log n) hashes.
Tree (RFC 9162 section 2.1.1)::
leaf hash = SHA256(0x00 || canonical_json(record))
node hash = SHA256(0x01 || left || right) # ordered
MTH([]) = SHA256("")
MTH([d0]) = leaf hash of d0
MTH(D[0:n]) = node(MTH(D[0:k]), MTH(D[k:n])) # k = largest power of 2 < n
The bottom-up build in _root (pair neighbours, carry a lone last node up
unchanged) gives exactly this tree. A lone node is never paired with a copy
of itself: that is the CVE-2012-2459 bug, where n leaves and n+1 leaves (a
copy of the last one appended) give the same root.
Proofs bind the position. A verifier derives the left/right order of every
step from leaf_index and tree_size (RFC 9162 section 2.1.3.2), so a
proof for leaf 1 does not verify as leaf 3, and a proof only verifies against
the root of the tree size it was made for.
File format: one record per line, json.dumps(record, sort_keys=True,
separators=(",", ":")) followed by \n. The leaf hash is taken over
exactly those bytes, so a line that is not in this canonical form is
rejected when the file is loaded (it would let two readers see different
content under the same hash, for example with duplicate keys).
Crash safety: writers take an exclusive flock on the file, write one
full line and fsync it. Readers take a shared lock. If the process dies
in the middle of a write, the last line can be incomplete. Loading such a
file raises :class:TornTailError; repair_log (CLI: repair) moves
the broken last line to <file>.corrupt. A broken line anywhere else
raises :class:LogCorruptError and cannot be repaired: it means the file
was changed.
MerkleLog ¶
An append-only Merkle log of violation records.
The leaves are kept in memory; an append-only file mirrors them on disk. Several processes may append to the same file: each append locks the file, reloads it if another process changed it, writes one line and fsyncs it.
append ¶
Append a record; return its leaf index.
The record is serialized and hashed before anything changes, so a
record that cannot be stored leaves the log as it was. The log keeps
its own copy: changing record afterwards has no effect.
With unique=True a record whose (trace_id, span_id) is already in
the log is skipped and None is returned.
inclusion_proof ¶
Audit path for leaf_index in the tree of the first size
leaves (default: all of them).
consistency_proof ¶
RFC 9162 proof that the tree of size leaves (default: all)
extends the tree of old_size leaves.
LogCorruptError ¶
Bases: ValueError
A line of the log file is broken. In the middle of the file this means the file was changed by something other than this code.
TornTailError ¶
Bases: LogCorruptError
Only the last line is broken, which is what a crash during a write
leaves behind. repair_log moves it aside.