The profiler reads each source and learns its shape before a single rule exists: fill rates, formats, value lists, distributions, key candidates.
For each column the profiler records the fill rate, the formats seen and their frequencies, the distinct values and the long tail, the numeric distribution, the dates' range and the relationship to other columns. From these it computes a score per dimension: completeness from the fill rate, validity from the formats against the expected pattern, consistency from the row relationships, uniqueness from the key candidates, timeliness from the dates.
Accuracy needs the world, so the profiler samples: it verifies a sample of tax ids, addresses and bank accounts through the verification services and extrapolates a score with its confidence interval. The full verification runs later as a service agent.
The result is the scorecard: one number per column per dimension, with the failing values listed so the steward can see what a failure looks like before writing the rule for it.
| column | fill | formats | score | top issue |
|---|---|---|---|---|
| STCD1 (tax id) | 96% | 3 | 96% | 1,672 blank on active |
| NAME1 | 100% | — | 92% | case and suffix variants |
| LAND1 | 99% | 4 | 74% | “United Kingdom”, “UK”, “GB” |
| SMTP_ADDR | 70% | 2 | 61% | 12,540 blank |
| TELF1 | 58% | 9 | 38% | nine formats |
| BANKN | 88% | 2 | 88% | 312 unverified > 12 months |
This view feeds survivorship directly: the winning source per field is a starting proposal for the attribute group's rule.
| Field | SAP | CRM | Registry | Proposed winner | Why |
|---|---|---|---|---|---|
| Registered name | 92% | 78% | 100% | Registry, then SAP | The registry is the world; SAP is closest to it |
| 61% | 89% | — | CRM | Filled and well formed far more often | |
| Country | 74% | 91% | 100% | Registry, then CRM | SAP holds free text; the registry holds the code |
| Credit limit | 96% | 94% | — | SAP | Finance maintains it in the ERP |
| Phone | 38% | 62% | — | Verification task | Neither source is reliable; verify or drop from the golden record |
Where every source is weak, the profiler proposes a verification task or a decision to leave the field out of the golden record. A golden record with fewer, trustworthy fields beats one with every field and no confidence.
Scores are kept per day, so the effect of a rule is visible as a slope: the country standardisation rule deployed on the showcase estate took the source's consistency from 54% to 81% in three weeks, and the trend shows the day it went live. A rule with no slope is a rule to revisit.
Cost translates a score into money for the owner: the field weight says what one failure costs, the consumer weight says how many processes read it, and the count of failing records multiplies them. Missing carrier codes on 31% of dispatch records cost a consumer-goods company £420,000 a month in misattributed freight in the illustrative estate; the number is what put that field first.
The cost view orders the first week's work: the ten costliest fields get rules on Tuesday. Everything else waits until they are under control.
Country standardisation on the showcase supplier source, three weeks after go-live.
Carrier codes blank on 31% of dispatch records, in the illustrative consumer-goods estate.
The costliest ten by weight × failures, from the profiler's own numbers.
Fields no source holds well become verification tasks or leave the golden record.
It reads in place through the connector and keeps the statistics and the sampled failing values; the rows stay in the source. Where a source is too slow to profile live, the Backbone lands a governed copy and the profiler reads that.
A source of half a million rows profiles in under two hours on the showcase estate; the cross-source comparison runs once the match keys are set, which is the same afternoon.
The steward reads the column view to write rules; the owner reads the cost view to set priorities; the governance lead reads the trend; the consumer reads the contract view on the enforcement page.
Bring one domain and two systems. In a week the sources are profiled, the ten costliest fields are under rules, the first corrections are written back with their acknowledgements and the publish gate is on.
