Reviews
Part of Marketing attribution reviews: a gated test framework
Reviewing an attribution claim by evidence level, from supplier record to buyer test
Review one marketing attribution claim by evidence level: current supplier record, signed contract, independent record and a controlled buyer test, kept apart.
What to take away
- Review one claim at a time and give each piece of evidence one level.
- Level 1 is a current first-party record, level 2 a signed contract, level 3 an independent record and level 4 a controlled buyer test.
- No level substitutes for another, and levels are never added into a score.
- A platform-credit check can reproduce a documented rule. It cannot establish causal lift.
- A claim with no level behind it stays unresolved.
Four levels answer four different questions
Each evidence level answers a different question, so a claim needs the level that matches what it asserts. A supplier page says what the supplier states today. A contract says what the supplier is bound to. Only a test shows how the service behaves on the buyer's data.
| Level | Evidence | What it can show | What it cannot show |
|---|---|---|---|
| 1 | Current first-party record | What the supplier states about the exact service and edition today | That the service performs as stated |
| 2 | Signed contract | What the contracting entity is bound to provide | That the output is accurate |
| 3 | Independent record | What a party with no stake in the sale has recorded | That the result holds for this buyer's data |
| 4 | Controlled buyer test | How the service behaves on the buyer's frozen fixture | Anything beyond the job, data and dates tested |
Four Evidence Levels
Evidence level
- First-party record
- Supplier statement
- Signed contract
- Enforceable obligation
- Independent record
- Specific issue
- Controlled buyer test
- Declared fixtures
What it can establish
- First-party record
- Enforceable obligation
- Signed contract
- Causal performance
- Independent record
- Full platform behaviour
- Controlled buyer test
- General causal lift
What it cannot
- First-party record
- Signed contract
- Independent record
- Controlled buyer test
None of these levels substitutes for another. A strong contract does not turn a level 1 performance claim into a tested one, and a test does not replace the contract. A review expires if the candidate edition, legal entity, data flow, model or material contract term changes.
Grade one claim at a time
Write the claim exactly as the supplier states it, then grade the evidence behind it. Do not review a service as a whole, because a single verdict hides which claims have support and which have none.
- Copy the claim in the supplier's own words, with its date and source.
- Name the attribution job it belongs toplatform credit allocation, cross-system reconciliation, descriptive analysis, marketing mix modelling, or experiment and incrementality support.
- List each piece of evidence and give it one level.
- Record the highest level reached and the level that is missing.
Choose one job per review, because combining jobs would make apparent feature differences meaningless. The protocol names the decision owner, eligible data, source versions, expected output, uncertainty requirement and rejection conditions. It also identifies the existing process that the claim is measured against.
CAP's substantiation guidance explains that objective marketing claims need documentary support and that testimonials alone are unlikely to prove an objective claim. The same caution applies when a reviewer reads a supplier case study, which stays at level 1 until something stronger backs it.
Earn level 4 with a frozen fixture
A level 4 result exists only when the test can be repeated. Use synthetic or specifically authorised data, and freeze event names, identity rules, eligible population, date range, Europe/London timezone and query or model version before anything runs.
Freeze the treatment of missing, duplicate, late, unmatched, refunded and corrected records too. For each task, record the expected state before the test and the observed state afterwards, so a reviewer can compare them without relying on memory.
Illustrative fixture: event names order_complete, lead_submitted and session_start; identity rule is a deterministic customer_id match with hashed email fallback, no probabilistic matching; date range 1 January 2026 to 31 March 2026 in Europe/London timezone. This shows the level of detail to freeze; it is not a real client dataset.
Freeze Before Testing
- Event names
- Identity rules
- Eligible population
- Date range and timezone
- Query or model version
- Missing, duplicate, late records
- Refunded and corrected records
Keep causal language out of a descriptive result
A platform-credit check can confirm that a documented rule was reproduced. It cannot establish causal lift. A successful level 4 test of reproduction therefore supports an allocation claim and nothing stronger.
HM Treasury's QPIE framework describes the importance of an appropriate counterfactual for causal attribution in government evaluation. Apply that boundary when examining commercial performance claims. A causal claim without a counterfactual stays unresolved, however many records are offered for it.
Separate gates sit beside the evidence levels: data protection and PECR, security, accessibility, contract, full cost and exit. Keep them separate and do not average them.
Where profiling or other processing may create high risk, the buyer should screen the actual configuration with a qualified reviewer. The ICO's DPIA guidance covers how to decide whether to do a DPIA. The launch review sets out the record a launch needs.
Report what is graded and what is not
Report the graded claim, the level reached and the level still missing. The report should list admitted evidence, unresolved claims, test results, exceptions and expiry, so a reader can see exactly where the evidence stops.
Scores, weights and pass thresholds remain blank until accountable buyer reviewers set them. A failed security gate cannot be cancelled by a preferred interface. Name untested areas such as ingestion, identity matching, output accuracy, permissions, support, recovery, export and deletion as untested rather than leaving them out. An empty dossier gets no verdict, as in agency reviews.
Choosing between candidates is a separate step, laid out in the selection checklist. A graded claim feeds that choice as one input and never replaces it.
Before you act
- Copy the claim in the supplier's own words, with date and source.
- Name the one attribution job the claim belongs to.
- Give each piece of evidence a single level from 1 to 4.
- Freeze event names, identity rules, timezone and query version before testing.
- Record the expected state before and the observed state after each task.
- List unresolved claims, exceptions and expiry in the report.
Common questions
Can a signed contract prove that a service performs as claimed?
No. A contract is level 2 evidence of what the contracting entity is bound to provide. Whether the output is accurate on the buyer's data is a question for a controlled buyer test, which is level 4.
What must be frozen before a test is run?
Event names, identity rules, eligible population, date range, Europe/London timezone, query or model version, and the treatment of missing, duplicate, late, unmatched, refunded and corrected records. Only synthetic or specifically authorised data should be used.
What can a platform-credit check establish?
A platform-credit check can confirm reproduction of a documented rule, but it cannot establish causal lift. That boundary applies when examining commercial performance claims.
