Outlook
Part of A monitoring framework for a marketing attribution outlook
Seven steps to test AI in marketing attribution on known cases
Test an AI task in marketing attribution on frozen known cases: the reference answers, the scoring of errors, the acceptance criteria and when to re-run.
Marketing attribution AI applications should be tested on cases whose correct answer is already known, not judged by how fluent the output sounds. "Use AI for attribution" is too vague to test, so the test starts with one small task and a human owner.
No system was accessed, and no claim is made about accuracy, time saved or 2027 adoption in England. The AI Playbook for the UK Government, published on 10 February 2025, offers guidance on using AI safely, effectively and securely in government. It is not a rule or performance record for private marketing teams.
What to take away
- Define AI attribution as a small task with a named human owner and a manual baseline.
- Freeze known cases first, and have the owner write the reference answers by hand.
- Test for error classes such as false positives, false negatives and subgroup effects, not fluent wording.
- Agree acceptance criteria before the task is run, and fail the task if any criterion is missed.
- Keep production on hold until the human owner can reproduce, reject, correct and roll back output.
- Re-run the test at every model, prompt, data or supplier change and when the governing source changes.
Seven steps to a reproducible test
A test is reproducible only when it runs on cases whose correct answer is already known, so the reference set comes first. The seven steps fix the owner, the cases and the criteria before any output is seen.
- Freeze a sample of past attribution cases (journey and touchpoint records exported from a platform such as GA4 or Adobe Analytics, or from a mobile measurement partner such as AppsFlyer, with the reports built from them) and lock the cut-off date.
- Have the named human owner produce the reference answer for every case by hand, using the definitions the team already works to: the conversion window, the channel taxonomy, the credit rule and the reconciliation method.
- Run the AI task on the same cases without giving it the reference answers, and without letting it infer an identity to complete a journey.
- Score every output against the reference, recording false positives, false negatives, missing cases, subgroup effects and reviewer disagreement per case and per subgroup.
- Apply the acceptance criteria that were agreed before the run, not ones chosen after seeing the results.
- Fail the task if any criterion is missed. A pass admits it to a controlled trial only.
- Re-run the whole test whenever the model, prompt, data population, supplier or governing source changes.
What to count for each task
Each task fails in its own way, so count a different error for each. The table pairs every task with the output a trial may produce, the error to count and who reviews it. Decision authority stays with an accountable person by default.
| Task | Permitted trial output | Error to count | Review |
|---|---|---|---|
| Generated explanation | Draft wording about an existing, frozen report | A misdescribed metric stated with certainty | Verify every figure, limitation and causal label |
| Classification | Proposed label under a buyer-owned taxonomy | Inconsistent or wrong labels | Check ambiguous, missing and misclassified records |
| Anomaly triage | Queue of unusual observations | A quiet but systematic defect that is missed | Analyst investigates source and decides action |
| Modelling assistance | Code, specification or diagnostic suggestion | Reproducible code that answers the wrong causal question | Statistician reproduces and validates the method |
| Decision authority | None by default | Not tested, because the model does not decide | Accountable person decides budget, targeting or publication |
Bounded AI tasks and review
Task
- Generated explanation
- Draft wording
- Classification
- Proposed label
- Anomaly triage
- Queue of observations
- Modelling assistance
- Code or diagnostic
- Decision authority
- None by default
Permitted trial output
- Generated explanation
- Verify figures and labels
- Classification
- Check ambiguous records
- Anomaly triage
- Analyst investigates source
- Modelling assistance
- Statistician reproduces method
- Decision authority
- Accountable person decides
Human or statistical review
- Generated explanation
- Classification
- Anomaly triage
- Modelling assistance
- Decision authority
Generated prose can sound certain while misdescribing a metric. Classification can hide inconsistent labels, and a modelling suggestion can reproduce code yet answer the wrong causal question. A trial uses synthetic or specifically authorised fixtures.
Write the acceptance criteria before the run
Agree the acceptance criteria before the test, not after. Set the share of false positives, false negatives and missing cases the owner will tolerate, and the level of reviewer disagreement at which the task counts as not testable yet.
The ICO's AI accuracy guidance says the appropriate level of statistical accuracy of a generative AI model is linked to the specific purpose it is used for. Set each criterion for the task and its consequence. Qualified review must confirm current law and its application to the actual configuration.
Fail the task if any criterion is missed. Passing admits it to a controlled trial only, and confidence in any benefit remains unknown. Production stays on hold until the owner can reproduce, reject, correct and roll back the output.
Judge the output against the team's own definitions
Score each output against the definitions the team already uses for incrementality testing, marketing mix modelling (MMM) and multi-touch attribution. A generated explanation that reproduces a platform number but contradicts the incrementality read or the MMM is an error, not a pass.
Platform credit, reconciliation, modelling and causal incrementality each need their own definitions and evidence. Record false positives, false negatives, missing cases, subgroup effects, correction routes and reviewer disagreement, so a reviewer can see which cases failed and why.
Do not let the model infer an identity merely to complete a journey. Keep that rule in the test script itself, not only in a policy document.
Protect inputs and set the stop rules
Protect models, data and prompts with versioning, access control, audit records and restoration before the controlled trial starts. These controls do not certify a supplier, and they sit beside the error test rather than replacing it.
UK GDPR and PECR questions remain separate from model quality. If an attribution workflow uses tags, pixels or scripts, those rules apply to the actual purpose and configuration, and AI capability does not provide legal permission.
Accessibility testing should cover the generated explanation, exception queue and correction path. Contract review should address input use, model training rights, logs, sub-processors, changes, export, deletion and termination. Security and legal reviewers need authority to stop deployment.
No forecast or announced attribution-specific AI change is relied upon here. The buyer scenario expires if the task, model, data population or governing source changes. Production remains on hold until the human owner can reproduce, reject, correct and roll back the output.
Before you act
- Name the model, version, data, expected output, error classes and manual baseline.
- Freeze the fixtures and build the manual reference answers before running the task.
- Agree acceptance criteria (tolerated false positives, false negatives and missing cases) before the test.
- Record false positives, false negatives, missing cases, subgroup effects and reviewer disagreement.
- Confirm UK GDPR, PECR and accessibility for the actual configuration.
- Review contracts for input use, training rights, logs and deletion.
- Give security and legal reviewers authority to stop deployment.
- Reproduce, reject, correct and roll back the output before production.
Common questions
Why test on known cases rather than judge the wording?
A test is reproducible only when it runs on cases whose correct answer is already known. The named human owner writes the reference answers by hand before the AI task runs, so each output is scored against something fixed rather than against how convincing it sounds.
When are the acceptance criteria set?
Before the test. The owner agrees the share of false positives, false negatives and missing cases to tolerate, and the level of reviewer disagreement at which the task counts as not testable yet. Criteria chosen after the results would let the outcome set its own bar.
What does passing the test allow?
Passing admits the task to a controlled trial only. Production remains on hold until the human owner can reproduce, reject, correct and roll back the output. The test is re-run whenever the model, prompt, data population, supplier or governing source changes.
