sharedata — The Protocol, In Full

The shape of it

Two phases, one key idea.

The protocol splits into a batch matching phase that runs hours or days ahead of time, and a real-time serving phase that answers in nanoseconds. Splitting them is what lets the cryptography be strong: the expensive work happens off the hot path, so nothing has to compromise to fit inside a 100 ms auction window.

The organizing principle: raw identities never move. A buyer turns its targets into opaque tokens; each seller derives the same function over its own database, on its own hardware, and ships back only an unforgeable proof for the records it actually has. Match computation is decentralized to the data, not the data centralized to the computation.

None of this is advertising-specific. We use a buyer/seller framing below because it's concrete, but the two parties can be any data holders — a fraud co-op and a merchant, two lenders, a hospital and a registry. Anything you can reduce to a canonical key, you can match on.

It's just keys

What you can match on.

The protocol doesn't care what an identifier means — only that both sides can normalize it to the same canonical byte string. That's the whole flexibility: one field or a composite of many, the token is computed identically.

Record type	Fields you key on	What it unlocks
Traffic	ip (+ user-agent)	Dedup a visitor stream, or enrich it against a partner's IP risk/intent signal.
Emails / device	email · phone · maid	Audience overlap, suppression lists, identity resolution.
Leads	name · address · phone · email · location	Match and dedup lead lists across partners without exposing either list.
Transactions	name · address · phone · location	Tie purchases to an audience, or check a transaction against a fraud set.

A single field is the simple case. For a multi-field record — a lead or a transaction — you normalize each field and concatenate them into one canonical key, so the whole record collapses to a single token exactly like an email would. (Partial matching, e.g. name+phone or name+address, is just several derived keys checked in parallel.)

# single identifier key = normalize(email) # "ada@example.com" key = normalize(ip) # "203.0.113.7" # composite record (a lead or transaction) key = normalize(name) || normalize(phone) || normalize(zip) || normalize(addr) T = HMAC(k, key) # same token, any record type

The keyspace decides the primitive — and this is the part you must get right. High-entropy keys (emails, full composite leads/transactions) are safe under plain HMAC: the space is far too large to brute-force. But small or enumerable keys — a bare IPv4 (~4 billion values), a phone area, a lone ZIP — could be swept by whoever holds the key. For those the config layer auto-selects an OPRF (oblivious PRF) flow with rate-limiting, so no party ever holds the full key alone and neither side can enumerate the space. Choosing HMAC-vs-OPRF correctly is the safety rail — and the platform decides it for you, so a non-expert can't footgun a leak.

Cast & symbols

Roles and notation.

Buyer — holds the target list, mints campaign keys and target tokens, verifies proofs, activates bidding.
Token distribution service — routes opaque token sets buyer → sellers. Cannot decrypt, reverse, or interpret them. Never touches a user database, never sees a match.
Seller / publisher — holds a user database, runs matching locally, emits proofs of possession for matched users only.

# notation used throughout HMAC(k, m) keyed-hash (SHA-256), one-way without k normalize(ID) canonicalize: lowercase, trim, standard form || byte concatenation k 256-bit campaign-scoped key N / M #target identities / #users in seller DB

Phase A · asynchronous

Pre-auction matching.

Runs decoupled from real-time bidding, so there are no latency constraints on the crypto. This phase produces the token set every seller will match against.

Step A1 · buyer

Campaign key generation

For each campaign the buyer draws a fresh key from a CSPRNG. It is unique per campaign and destroyed at campaign end — that scoping is what kills cross-campaign linkage later.

k ← random(256 bits)

Step A2 · buyer

Target token generation

Each target identity becomes a token T. The output looks like a random 256-bit value and reveals nothing about the identity without k.

T = HMAC(k, normalize(ID_target))

Step A3 · buyer → service → sellers

Token distribution

The buyer ships the token set plus a campaign_id through the distribution service to participating sellers. The service forwards opaque bytes; no target identities are ever transmitted.

Phase B · publisher-local

Proof of possession.

Everything here happens inside the seller's infrastructure boundary. No user data leaves the seller's environment at any point in this phase.

Step B1 · buyer → seller

Key receipt

The campaign key k reaches each participating seller over a secure channel. (Alternative embodiments derive it via Diffie–Hellman, removing explicit transmission.)

Step B2 · seller

Token set indexing

The seller builds a hash table over the token set for O(1) lookup. A 50M-target campaign is ~1.6 GB at 32 bytes/token — O(N) memory, O(N) build.

Step B3 · seller

Parallel database scan

For each of its M users the seller derives the same function and probes the table. Every HMAC is independent — embarrassingly parallel across cores, servers, or GPUs.

for each ID_u in seller_db: T_u = HMAC(k, normalize(ID_u)) if T_u in token_table: # match emit_witness(ID_u)

Step B4 · seller

Witness generation

A match isn't an assertion — it's a proof. The witness W binds the identity to the campaign, so it can't be replayed elsewhere, and can't be produced without actually holding the user.

W = HMAC(k, normalize(ID_u) || campaign_id)

Step B5 · seller → buyer

Proof transmission

Sellers with matches return the witness set; sellers without send nothing. Forging a valid witness would require inverting HMAC — infeasible.

It has to be cheap

Scale & throughput.

Cost is linear: a seller computes M HMACs per campaign, C × M for C concurrent campaigns. Reference workload below is 80M users × 2,000 campaigns = 160B HMAC operations per full cycle.

Configuration	Throughput	Full cycle
64-core CPU server	~64M ops/s	—
Cluster of 10 CPU servers	~640M ops/s	~4 min
Single NVIDIA H100	~7B ops/s	~23 s
8-GPU node	~56B ops/s	~3 s

Incremental updates. You rarely rerun the full cycle. A new user is C HMACs (one per active campaign) — negligible. A new campaign is M HMACs against the full DB — ~11 ms for 80M users on a single GPU. Between incremental updates and periodic recompute, the match cache is never more than minutes stale.

The hot path

Real-time serving via Bloom filters.

The batch phase yields a match table: per user, the campaigns it matched. To serve that at impression time, the publisher compiles each campaign's matched IDs into a Bloom filter for the edge.

5M matched IDs at a 0.1% false-positive rate ≈ 9 MB; 2,000 campaigns ≈ 18 GB — fits on one edge server or partitions across nodes.
One probe ≈ 200 ns; 2,000 campaigns ≈ 0.4 ms total — invisible inside a 50–100 ms header-bidding window.
The ~0.1% false positives fall back to a full cache verification at negligible cost.

No identifier in the hot path. At impression time there is zero HMAC computation. The lookup uses a domain identifier the publisher already holds — it's a memory probe, not a crypto operation.

Phase C · buyer-side

Verification & activation.

Step C1 · buyer

Proof verification

The buyer recomputes the expected witness and compares. Equality is cryptographic certainty that the seller holds the target.

W_expected = HMAC(k, normalize(ID_target) || campaign_id) verify: W == W_expected

Step C2 · buyer

Qualified seller set

Sellers with valid proofs form a campaign-and-target-specific qualified set.

Step C3 · buyer

Bidding activation

In subsequent auctions the buyer restricts bids to the qualified set — spend lands only on inventory that can demonstrably reach the target.

Step C4 · buyer

Key destruction

At campaign end k is destroyed. Past witnesses become useless for any cross-campaign tracking.

Follow the bytes

What actually crosses the wire.

The publisher's user database never leaves its infrastructure. Concretely:

Never leaves the publisher

The raw user database & email addresses
Every non-matching user record
The HMAC derivation & match checking
Bloom filter construction

Crosses the wire

Inbound: opaque token set + campaign_id + key k
Outbound: witnesses for matched users only
Each witness proves possession, reveals no identity

No external party — buyer, distribution service, or network operator — touches the user database at any step. This is architectural, not a policy promise: there is no step in the protocol where user data is transmitted externally.

The other pipe

Metering & settlement.

If raw data never crosses the wire, how does anyone get paid for it — and how does the operator take a cut without becoming the honeypot it just designed away? By splitting the system into two pipes that never touch:

The data pipe

Raw records, keys, blinding factors
Stays inside each party's trust boundary
Operator is never present here

The metering pipe

Usage counts, signed receipts, settlement
Operator is present here
Sees that a query happened, never what it was

It's the Stripe arrangement: present in the billing flow, absent from the product. The trick is making the count un-gameable without putting the operator on the live path — which is what pre-minted metering tokens do.

Step M1 · operator → client

Mint metering tokens

The operator mints signed, single-use metering tokens to the client — prepaid or against a credit line. Minting requires the operator's private key, which never ships in the open-source client, so reading or forking the code doesn't let you print your own.

mtoken = Sign(sk_operator, {nonce, credit, expiry})

Step M2 · client → provider

Spend one per query

Each query carries one token. The provider verifies the operator's signature offline — no live call to the operator, no added latency, no availability bottleneck — and only then serves the response. An honest provider refuses any query without a valid token.

Step M3 · client + provider

Tri-party receipt

The receipt is now bound three ways: the client signed the query, the provider signed the response, and the operator's signature rides in via the token it minted. Neither party can deny or misreport the query to the other or to the operator.

receipt = { Sign(client, query), Sign(provider, response), mtoken } # 3 signatures, 0 data

Step M4 · provider → operator

Redeem & settle

The provider batches spent tokens and redeems them with the operator. That redemption is when usage is counted and the take-rate is applied — asynchronously, off the live path. Actual money movement rides on Stripe Connect or similar; the novel piece is the verifiable count, not moving dollars.

Why it's hard to game. Tokens can't be forged (minting is cryptographic, not a strippable check). Usage can't be denied (the receipt is co-signed). A client editing its own software accomplishes nothing, because an honest provider still refuses tokenless queries. And silent underreporting shows up as a gap between tokens issued and tokens redeemed. The one residual case — a client and provider colluding off-book — is handled out-of-band by canary queries, defection bounties, and staked bonds, none of which touch the live path.

Two business models, sequenced. License the rails for a flat or tiered fee first (simple, no payments build, no disintermediation risk); layer the marketplace take on usage once there's liquidity. Throughout, the operator stays in the metering pipe and out of the data pipe — three signatures on every receipt, zero visibility into the bytes.

Why each step is there

Security properties.

Identity secrecy

T is a one-way function of the identity. Without k, the token reveals nothing recoverable.

Rainbow-table resistance

Campaign-scoped keys make precomputed tables useless — an attacker would need a fresh table per campaign across a 256-bit keyspace.

Proof unforgeability

Producing a valid witness without the identity means inverting HMAC or finding a collision — both infeasible.

Cross-campaign unlinkability

Unique, destroyed-after-use keys mean witnesses from different campaigns can't be correlated.

Replay protection

Binding campaign_id into the witness stops reuse across campaigns.

No trusted third party

The distribution service routes opaque bytes; no intermediary performs or can influence the match.

Honest about the edges

Scope & primitives.

This is tuned for the operations where cryptography is cheap: matching, membership checks, dedup, enrichment. General joint analytics over arbitrary SQL needs heavy MPC — that's where centralized clean rooms still win, and we cede it. The line: we do the checks and the matches; the clean rooms do the spreadsheets.

The hash function is swappable for any equivalent primitive — BLAKE3, SHA-3, KMAC, truncated HKDF-Expand. Acceleration can use GPUs, FPGAs, or ASICs given the embarrassingly parallel derivation. And the matching node can run inside a confidential-computing enclave (SGX, SEV, TrustZone) on the publisher's own hardware where attestation is required. The architectural invariant never changes: computation on the data owner's hardware, with their data, under their control.

The hard questions

FAQ.

The objections an engineer actually raises, answered directly.

If both sides compute HMAC(k, …), doesn't the key holder learn the other side's data? +

No raw data is exchanged — only tokens. The buyer learns which of its own targets a seller possesses (that's the point), never the rest of the seller's database. The seller learns nothing about which identity a token encodes. Each side only ever sees tokens for records it already holds.

Couldn't a party that holds k just brute-force the token space? +

Only if the keyspace is small. For high-entropy identifiers (emails, full composite leads/transactions) enumerating every possible input is infeasible. For small or enumerable inputs — a bare IPv4, a ZIP, a phone area — the config layer switches to an OPRF flow where no single party holds the full key, plus rate-limiting so neither side can sweep the space.

What exactly is the difference between the HMAC mode and the OPRF mode? +

HMAC mode shares a campaign-scoped key and both sides derive tokens independently — fast, simple, safe when the input space is huge. OPRF mode keeps the evaluation key split or held only by the provider, so the querying side gets a blinded evaluation without ever learning the key and the provider never learns the input. OPRF is the safe choice for guessable inputs; HMAC is the cheaper choice for unguessable ones.

How do you match a multi-field record like a lead or a transaction? +

Normalize each field to a canonical form and concatenate them into one key: normalize(name) || normalize(phone) || normalize(zip) || normalize(addr). The record collapses to a single token. If you want fuzzy or partial matching (name+phone or name+address), you derive several keys per record and check each in parallel.

What does “normalize” actually have to do? +

It guarantees both sides produce byte-identical input for the same real-world entity: lowercasing, trimming whitespace, stripping punctuation from phone numbers, canonicalizing email plus-addressing, standardizing address abbreviations, lowercasing/zero-padding ZIPs. Match quality is only as good as the normalization both sides agree on — it's the unglamorous part that determines hit rate.

What about typos and formatting drift — is this only exact match? +

The cryptographic step is exact-match on the canonical key, yes. Fuzziness is handled before hashing: aggressive normalization, plus emitting multiple candidate keys per record (e.g. with/without apartment number). Locality-sensitive techniques can bucket near-duplicates into shared keys, but anything approximate has to be designed into the key derivation — the HMAC itself is all-or-nothing.

Can a seller learn which of my targets it didn't match? +

It sees the full token set, so it knows how many targets exist and which ones it holds, but a token reveals no identity. It cannot recover the underlying identifier for a token it doesn't already have in its own database, because that would require inverting the keyed hash.

Doesn't sending the whole token set leak the size of my audience? +

Yes — set cardinality is visible by default. If that's sensitive, you pad the token set with random decoy tokens to a fixed size, or batch multiple campaigns together. The decoys never match anything, so they cost only bandwidth and a few extra lookups.

What stops a seller from just claiming it has a user it doesn't? +

The witness. A match returns W = HMAC(k, normalize(ID) || campaign_id), which the buyer recomputes and checks. Producing a valid witness without actually holding the identity means inverting HMAC — infeasible. “Yes” is no longer an assertion; it's a proof.

Can a witness from one campaign be replayed against another? +

No. The campaign_id is bound into the witness, and each campaign uses a fresh key that's destroyed afterward. A witness is only valid for the exact campaign it was generated under.

How is this different from a clean room? +

A clean room gathers both parties' data into one environment you both have to trust. Here neither database moves; each side computes on its own hardware and only tokens and proofs cross. There is no central pool to breach, subpoena, or misconfigure — the guarantee is architectural, not a policy in someone's terms of service.

Why not just exchange SHA-256 hashes of emails like everyone else? +

Unsalted hashes are trivially reversed with a precomputed table of common identifiers. Campaign-scoped keying defeats that: an attacker would need a fresh rainbow table per campaign across a 256-bit keyspace, which is infeasible. Plain H(email) is a privacy fig leaf; keyed derivation isn't.

How fresh are matches — is there a staleness window? +

Incremental updates keep it tight. A new user is one HMAC per active campaign; a new campaign is a full scan that finishes in ~11 ms for 80M users on a single GPU. Between incremental updates and periodic full recompute, the match cache is never more than minutes stale.

Where does the latency actually live? +

In the batch phase, which runs offline. The real-time path is a Bloom-filter probe (~200 ns each, ~0.4 ms for 2,000 campaigns) with zero cryptography in the hot path. That separation is deliberate: strong crypto off the critical path, a memory lookup on it.

What are the Bloom-filter false positives, and do they matter? +

At a 0.1% false-positive rate, roughly 1 in 1,000 lookups falsely reports a match and falls back to a full cache verification — negligible cost and no incorrect outcome, since the fallback resolves it. There are never false negatives: a real match is never missed.

If the operator never sees the data, how does it get paid? +

Through the metering pipe, which is separate from the data pipe. The operator mints single-use signed tokens, each query spends one, and providers redeem spent tokens for settlement. The operator counts usage and takes a rate without ever touching a record — the Stripe arrangement.

Why can't a client just print its own metering tokens? +

Minting requires the operator's private key, which never ships in the open-source client. Un-forgeability is cryptographic, not a check you can patch out by forking the code. Reading the source tells you how to verify tokens, not how to sign them.

Can a client and provider collude to dodge the take-rate? +

It's the one residual case, and it's handled out-of-band rather than on the live path: canary queries (the operator poses as a tokenless client; serving one is self-incriminating), defection bounties (the client earns more by reporting than colluding), and staked bonds that get slashed on detection. The goal isn't to make collusion impossible — it's to make the relationship that enables it irrational.

Does the metering token reveal anything about the query it paid for? +

No. The token carries a nonce, a credit amount, and an expiry — not the input, not the result. It proves “a paid query happened,” nothing about what was asked or answered. Three signatures on the receipt, zero visibility into the bytes.

What can't this do — when should I reach for something else? +

This wins where cryptography is cheap: matching, membership checks, dedup, enrichment. General joint analytics over arbitrary SQL needs heavy multi-party computation and is exactly where centralized clean rooms still win. The line: we do the checks and the matches; the clean rooms do the spreadsheets.

The protocol, in full.