Transaction data compared with reference data

Plain language meaning

Transaction data compared with reference data explains the difference between events that show what happened in banking activity and standing information that defines customers, accounts, products, parties, limits, calendars, schemes and control settings.

This topic is about banking data foundations for AI. It is not about treating every field as equal or mixing event data and master data without ownership.

In a real bank, this is not a loose technology choice. It affects customer outcomes, operational queues, payment execution, risk decisions, regulatory evidence, audit replay, privacy obligations and production resilience. AI should improve decision support and operating quality, but it must remain inside clear banking ownership and control boundaries.

Where it sits in the banking AI journey

This card belongs to AI Data and Model Operations. The working flow is Transaction event, Reference master, Join and validation, Feature creation, and AI-ready data set.

Read the flow as a bank operating model. Every stage needs a business owner, a source system, a data definition, a timing rule, an exception path, a fallback option, a monitoring requirement and retained evidence. That is the difference between a useful AI pattern and an uncontrolled technology shortcut.

Banking data and evidence

The important data points are payment event, posting entry, customer master, account master, product code, limit profile, calendar, and scheme rule. These items matter because they can alter risk scoring, payment treatment, customer communication, operational priority, reconciliation status, compliance review, model monitoring and management reporting.

The evidence pack should include data dictionary, join result, quality report, master version, event lineage, feature snapshot, and reconciliation result. A strong bank can replay the journey from source data to transformed input, model or rule output, human action, system outcome and monitoring result. A weak bank only knows that a process ran.

Controls that make AI adoption safe

The core controls are master-data ownership, event timestamp, join-key validation, effective dating, data quality rule, lineage record, and reconciliation check. These controls stop AI from drifting away from banking purpose, approved policy, data governance, model-risk expectations, operational resilience, customer fairness and auditability.

The practical design should define what AI may recommend, what it must never decide alone, which deterministic rule remains authoritative, who owns overrides, how degraded service is handled and what evidence is retained. Without that, the bank may gain speed but lose explainability and control.

Data, architecture and resilience lens

AI in banking depends on the quality of the surrounding architecture. The model can only be as reliable as the data contracts, event meanings, feature definitions, API controls, batch controls, reconciliation rules, monitoring signals and fallback processes that feed and govern it.

A bank-grade design therefore connects channels, source systems, payment hubs, risk systems, data platforms, feature stores, model serving, policy engines, case tools, audit logs and reporting layers. It also defines degraded operation, recovery evidence and post-incident learning before production use.

Regulatory and governance lens

Federal Reserve SR 26-2, dated 17 April 2026, gives revised model-risk guidance for traditional models and non-generative AI models used by banking organisations, including development, validation, monitoring, change control and governance.

NIST AI RMF 1.0 uses Govern, Map, Measure and Manage functions for AI risk management, and NIST AI 600-1 adds generative-AI risk actions for source grounding, content provenance, data protection, security and human oversight.

BCBS 239 remains current for effective risk data aggregation and risk reporting, and the Basel Committee's January 2026 newsletter reiterates the importance of accurate, comprehensive and timely data capabilities in banks.

FFIEC Architecture, Infrastructure and Operations guidance expects financial-institution technology environments to be governed, resilient, secure, monitored, documented and aligned to business risk.

CPMI's February 2026 updated harmonised ISO 20022 data requirements show why consistent structured data matters for interoperable cross-border payment processing and monitoring.

CPMI-IOSCO Principles for Financial Market Infrastructures explain governance, comprehensive risk management, liquidity risk, settlement finality and operational reliability for payment, clearing and settlement systems.

OFAC's Framework for Compliance Commitments describes sanctions compliance programme components including management commitment, risk assessment, internal controls, testing and auditing, and training.

FFIEC BSA/AML examination guidance expects suspicious activity monitoring systems to be risk-based, explainable by management, periodically reviewed and independently validated where appropriate.

Diagram walkthrough

Read the diagram from left to right as Transaction event, Reference master, Join and validation, Feature creation, and AI-ready data set. It is a control map, not decoration. It shows how banking data, AI support, policy control, human accountability and audit evidence should connect.

Use it as a 30-minute study method. For each box, ask which system creates the data, which rule or model acts on it, what can go wrong, who can override it, how the fallback works, what customer or regulatory impact exists and what evidence proves the final state.

Most important mistake to avoid

The common failure is training or scoring AI with joined data that looks complete but uses stale reference data, wrong effective dates or ungoverned master-data definitions.

The correction is disciplined scope. Keep the topic anchored to banking purpose, prove the data path, make ownership visible, test failure behaviour, record the evidence and make the final outcome explainable without depending on memory or assumptions.

One payment, two kinds of evidence

A fictional bank receives a payment instruction for 750 units of a currency at 10:15. The transaction record describes an event: amount, currency, parties, channel, timestamps, status and a bank-generated identifier. Reference data gives stable meaning to fields used by many events: account ownership, product classification, branch, country code, currency definition, counterparty status or an approved code list. The distinction is not that reference data never changes. It is that a change must be versioned and effective-dated so an older transaction can be interpreted as it was at the time.

Suppose the customer changes address on Wednesday and an investigator reviews Monday's payment on Friday. Joining Monday's event to Friday's current address can create a false story about the original risk decision. The analytical view should retain the address or jurisdiction value available at the decision point, the source record and effective time. A later correction may be relevant to the investigation, but it should be recorded separately. The same principle applies to beneficiary master data, account status and sanctions-list versions.

An analyst should specify the join key, ownership and effective-date rule for each reference source. What happens when the transaction has an unknown currency code, an account identifier maps to two customer records, or the reference feed arrives after the payment? A default code such as "other" may be acceptable for a descriptive dashboard but unsafe for a control decision. The pipeline should distinguish invalid, absent and late values, quarantine or refer according to the approved workflow, and reconcile the number of records processed.

Do not teach a model that a field is reliable simply because it is populated. Compare the source's coverage, update cadence, correction history and permitted use with the transaction's decision deadline. A historical training set also needs the relevant reference-data version, not a join to today's master table. ISO 20022 external code sets illustrate why controlled codes can be updated or deactivated; their existence does not guarantee a bank's local mappings are correct. A reviewer should be able to trace the event, reference version, mapping and final action without treating a current-state lookup as historical truth.

Two kinds of facts in an AI decision

A banking transaction describes an event: a payment was instructed, a card authorization attempted, an account credited, a loan installment posted or a case disposition changed. Reference data describes the relatively stable entities, classifications and relationships needed to interpret events: a customer and account mapping, product code, currency, merchant category, branch, country code, payment purpose or legal entity. Neither category is perfectly static. Customers move, accounts close, product definitions change and counterparties are merged. The distinction matters because a model's feature value is often produced by joining an event at a specific time to the reference state that was valid at that time.

Consider a transfer instruction. The transaction record can contain an instruction ID, payer account, payee account, amount, currency, channel, creation time, acceptance time, status and eventual settlement or return. A customer master can associate the account with a legal customer and an effective date. A beneficiary directory can classify whether the recipient was previously known. A country reference can map an ISO code to a jurisdiction for a rule. A product reference can determine whether the payer account is eligible for a particular service. A model using recipient novelty or payment velocity needs all these facts with explicit time semantics.

The transaction is not necessarily a single row. An instruction can produce validation events, a pending status, a screening hold, release, settlement and later return. A card authorization and clearing record can differ. A loan disbursement and repayment are separate events linked to the same facility. AI training should not mistake each status change for a separate independent customer decision. Define the unit of analysis, its stable identifier and which event represents the observation time and outcome.

Effective time and record time

Reference data needs at least two temporal questions: when was a value valid in the real business domain, and when did the bank learn or store it? A customer address corrected today may have been valid since last month, but a decision made two weeks ago may only have seen the old address. For a historical fraud training set, using the corrected value as though it was known at authorization time leaks future information. Store effective-from and effective-to intervals where appropriate, plus ingestion or recorded timestamps, and define which clock a feature join uses.

An as-of join selects the reference record eligible at the event's decision time. It must handle gaps, overlaps and late corrections. If two customer-master records both claim to own an account on the same date, the pipeline should flag ambiguity instead of arbitrarily picking the last row. If a product code changed meaning after a merger, preserve versioned mappings. Analysts should be able to replay a past decision from the reference values available then, and also compute a corrected view for remediation. These are separate outputs.

Transaction event time and processing time also diverge. A mobile payment can be created offline and received later. A cross-border message can traverse systems before ledger posting. A fraud feature such as payments in the prior hour must specify whether the window is anchored to initiation, authorization or settlement. Streaming windows based on ingestion time can count late arrivals in a different period. The choice affects feature reproducibility and should be validated for each use case.

Identifiers and entity resolution

Banks often hold several IDs for the same customer, account or business across channels, acquisitions and legal entities. A transaction may carry a channel user ID, payment account ID, tokenized card ID or counterparty alias. A reference service maps these to governed entities. If the mapping is missing, a model may undercount prior behavior; if it falsely merges two people, it may contaminate a credit or fraud assessment. Entity resolution itself can use machine learning, but its confidence, override and effective date must be visible to downstream consumers.

For a fraud model, a card token and the underlying card account should not be conflated without understanding token rotation and shared cards. For AML network analysis, a legal entity, beneficial owner and authorized signatory are distinct relationships. For credit, a joint account and an individual borrower are not interchangeable. Use typed relationships and valid intervals. A generic customer ID join can hide both false links and missing links, especially when names or addresses are used as weak matching evidence.

Create a crosswalk with provenance: source system, source key, canonical key, relationship type, matching method, confidence where relevant, approval status and validity interval. It should be access controlled because crosswalks can expose sensitive identity links. When a false merge is corrected, reverse lineage should find features and decisions affected by the earlier relationship, rather than only fixing the current master record.

Quality dimensions differ

For transactions, check completeness of eligible events, duplicate instructions, amount and currency validity, status sequencing, balance or settlement reconciliation, event delay and missing outcomes. A payment file can be syntactically valid yet omit a partition. For reference data, check key uniqueness within effective intervals, coverage of active entities, code validity, timely updates, conflicting attributes and relationship accuracy. The quality gate should reflect the decision: a missing merchant category may be tolerable for one model but critical for another.

Reference tables are often joined to millions of transactions. A one-percent mapping error can affect a large decision population even if the reference table is small. Conversely, a high-volume transaction feed may have a small number of corrupt events that disproportionately affect a rare but important high-value segment. Report quality by source, time, product and affected decision volume, not just overall row percentages.

Reconcile counts across lifecycle stages. For payments, compare accepted instructions with payment-hub events, settled items, returned items and ledger entries using documented timing and exclusions. For card fraud, link authorizations to clearing and chargeback records with explicit matching rules. A model's confirmed fraud label may arrive weeks later, while a declined transaction never has the same downstream observation as an approved one. Missing labels are not evidence of legitimate activity.

Feature design from both sources

An event-derived feature might count outgoing payments in the last 24 hours or measure amount relative to a customer's usual pattern. A reference-derived feature might encode account age, product type or beneficiary relationship. A combined feature might count first-time payees in a region under a versioned country or counterparty mapping. Each definition should specify the entity, eligible event statuses, time window, cutoff, currency conversion, missing-data rule and reference snapshot.

Avoid using a final event status in a feature computed before the decision. A fraud model scoring an authorization cannot know a later chargeback or manual case outcome. A loan model at application time cannot use repayment behavior on the new loan. Likewise, a reference classification created during an investigation may leak the outcome if joined retroactively. A point-in-time training builder should replay only information available at the modeled decision time, then join labels from a later defined outcome window.

For velocity, decide whether failed or reversed payments count. Repeated failed attempts can indicate attack behavior, while counting technical retries as independent instructions can inflate the feature. Use business IDs to deduplicate attempts and preserve retry metrics separately. For customer tenure, specify whether the clock starts at first account opening, current product activation or verified relationship date. Each yields a different value and interpretation.

Currency conversion needs a rate source and timestamp. A model comparing payment amounts across currencies should not apply today's exchange rate to a historical decision without documenting that choice. Reference data for rates is time varying and may have multiple quote conventions. Fix the conversion rule in the feature contract, test edge currencies and handle missing rates explicitly.

Payment fraud example

At 11:02 a customer instructs a transfer to a new beneficiary. The payment hub emits the instruction with amount, channel and account IDs. The feature pipeline resolves the payer's governed customer key, joins the beneficiary directory as of 11:02, counts prior eligible transfers in a window ending just before the current instruction, and returns values with a freshness watermark. The model scores those values. The policy engine applies an approved threshold and separate mandatory checks before releasing or referring the payment.

If the beneficiary directory is updated at 11:05 to say the payee is trusted, the original 11:02 decision remains based on the earlier state. A replay for audit uses the old snapshot; a new payment at 11:06 may legitimately use the updated state. If the 11:05 change is a correction showing the payee was known all along, the bank can compute a corrected scenario and review affected decisions. It should not rewrite the historical evidence as if the correction had been available at 11:02.

Suppose the customer key cannot be resolved because of an upstream migration. Treat the velocity feature as unavailable, not zero. The model may have a validated missingness path or the orchestration policy may refer the payment. The log should preserve the source mapping error and resulting action. Monitoring should count affected instructions and assess customer delays as well as fraud exposure.

Credit example

A credit application record is a transaction in the broad sense: it captures a time-bound request and subsequent actions. Customer and product masters supply identity, relationship and policy context. A bureau response has its own observation date, received date, matching status and legal usage constraints. A cash-flow feature derives from account transactions under a defined lookback and exclusion policy. The model score, affordability rules and human decisions should each be recorded separately.

If two customers share an account, allocating all account inflows to one borrower can overstate income. The reference relationship should identify ownership and the feature should specify how joint activity is treated. A product migration can also change transaction codes for loan repayments, creating an apparent shift in delinquency if historical mappings are not versioned. Validation should inspect feature distributions around migrations and test individual reconstructed applications.

Labels need care. A default outcome belongs to a facility over a defined horizon, not every application event or every monthly score. A rejected application generally has no observed repayment outcome for that proposed loan, which creates selection bias. A training set that treats all missing outcomes as good accounts is invalid. Document the population and label policy with credit-risk owners.

AML and network example

Transaction monitoring may aggregate transfers between entities. Reference data supplies account-to-customer relationships, legal-entity hierarchy, beneficial ownership and sanctioned or high-risk classifications. Those relationships can be uncertain, dated and jurisdiction dependent. A graph model should distinguish an observed payment edge from an inferred ownership edge and retain provenance. Combining them as equivalent links can make a network look more suspicious than the evidence supports.

A later investigation may confirm a relationship that was only suspected at alert time. For model evaluation, retain what was known at the original decision, and record the later confirmation as outcome evidence. A sanctions-list update is a reference-data change with its own effective and receipt times; it must not silently backfill a past screening result. Mandatory screening procedures and model-generated AML priority should remain separate in the decision record.

Architecture and ownership

The transaction system owns the authoritative event and status lifecycle. Reference-data owners govern identifiers, classifications and relationship changes. A data platform can publish curated versions with contracts for key meaning, validity periods, quality and access. A feature platform owns the implemented transformation and online/offline consistency; it should not quietly redefine upstream business codes. Model owners approve which feature versions are used. The business decision owner approves how missing or conflicting data changes the action.

Use event schemas with stable IDs and explicit time fields. Use slowly changing reference records or another versioned representation when historical joins are required. Publish change events or snapshots with watermarks and a reconciliation process. A stream of updates alone may be insufficient for replay if it omits deletes, corrections or initial state. A snapshot alone may be too stale for a real-time decision. Choose the combination that matches the decision latency and audit requirements.

Access controls follow purpose. Raw payment narratives and customer identifiers can be sensitive. A model may need derived features and controlled entity references, while investigators need authorized drill-down. Encrypt data in transit and at rest, log access, set retention periods and review vendor transfers. The fact that a value is labeled reference data does not make it public or safe to copy widely.

Validation exercise

Choose a sample of payment, credit and AML decisions. For each, identify the event that triggered the decision, its exact timestamp, every transaction status used, every reference version joined, any identity match and the later label or disposition. Replay the feature computation as of the original decision and compare it with the stored value. Then simulate a late event, a corrected customer mapping and a changed product code. Record which historic decisions change in a corrected analysis and which original records remain immutable.

Test population-level controls too: uniqueness of business IDs, one active reference record per effective interval where required, completeness by source and partition, currency code validity, event-to-ledger reconciliation and feature freshness. Quantify affected decisions when a check fails. A green pipeline execution is insufficient if the wrong transactions or reference versions entered the model.

The practical distinction is not that transactions move and references never move. It is that they have different ownership, update patterns and time semantics. Banking AI needs both, joined with explicit evidence of what was known when each decision was made.

Common join failures and their remedies

A one-to-many join can multiply transactions. Suppose a customer has three active address records because the master system did not close previous versions. Joining 10,000 payments to the table may produce 30,000 rows. A training pipeline can then overweight that customer, inflate velocity counts and distort fraud labels. Count rows before and after each join, assert permitted cardinality and inspect duplicate keys. Selecting an arbitrary address merely conceals the data defect. Determine the governing address rule and repair or quarantine ambiguous records.

An inner join can silently delete the very transactions a model most needs to examine. New customers may lack a complete reference profile, and suspicious counterparties may not resolve to a known entity. If the pipeline keeps only matched rows, evaluation is biased toward established relationships. Use explicit unmatched categories, measure match rates by segment and period, and define a safe model or policy path for missing relationships. A missing join is not automatically evidence of risk, but it is an important observation.

A many-to-one mapping can be wrong across legal entities. Two banks in a group may reuse local account identifiers. Joining on account number without entity or system scope can attach a transaction to the wrong customer. Composite keys and source namespaces should be specified in the contract. Test collisions at ingestion and after mergers. When mapping logic changes, evaluate how many historic decisions would have received a different feature value.

Late-arriving reference updates can create inconsistent online and offline values. An online service might use a current beneficiary classification while an offline training table builds features from a daily snapshot. Compare both paths on the same decision IDs and event cutoffs. Differences should be explained by documented latency and version rules. If the model was trained with a feature whose live meaning differs, a successful deployment test does not establish performance in production.

Status-code reuse is another trap. A payment code such as "R" may indicate returned in one source and referred in another. A product migration can change its meaning while retaining the code. Map source and version together with code; test known examples with business owners. Free-text narratives are not a reliable substitute for a status contract, even if an LLM can summarize them. The source meaning must govern the decision.

Exercise: construct a point-in-time feature

Define the feature "number of distinct newly added beneficiaries paid in the prior seven days." Begin with an eligible payment event set: successful or accepted instructions according to the business definition, excluding duplicates and technical retries. Define a customer key and a decision timestamp for the payment being scored. Exclude the current instruction and future events. Resolve each beneficiary identity through the versioned mapping available at the prior event time, and determine whether that beneficiary was new relative to the payer before that event. Count distinct beneficiary entities, not aliases, within a seven-day window ending immediately before the current decision.

Now challenge the definition. What happens if the bank learns after a payment that two beneficiary aliases represented one entity? For replay of the original decision, preserve the earlier identity view. For a corrected analytic view, merge them and label the correction. What if a prior instruction was accepted but returned three days later? The online feature could not have known the return at the earlier decision. The offline training builder must reproduce that knowledge boundary or explicitly justify a different feature. What if a beneficiary was saved but never paid? The feature says paid, so saving alone does not count.

Test six records by hand: a first payment to A, a retry of the same instruction, a second payment to A, a payment to B, a payment to an alias of A and a returned transfer. State each event's time, status, source key and mapping version. Calculate the feature before and after each eligible event. Use this hand-checked set as an executable test for the batch and online implementations. The goal is not a clever SQL expression; it is agreement on the event and reference semantics.

Exercise: measure reference drift

Take a product-code reference that maps source products into lending, deposits and payments. For each month, count active customer accounts by source code and mapped category. Report unknown codes and accounts assigned to multiple categories. Compare the mapping version used in training with the live version and measure the number of model requests whose product feature changes. If a new product appears with no mapping, decide whether the model should abstain, use a validated unknown category or route to manual policy. Do not silently map it to a familiar product merely to keep the pipeline green.

Do the same for country or jurisdiction reference where relevant. Codes can be missing, obsolete or ambiguous; geographical risk classifications may change under policy. A model's learned association with a category is not a substitute for current legal requirements. Keep source country codes, policy classifications and model features distinct. A policy change should be visible as a policy version, while a statistical feature change should be governed through the model and feature lifecycle.

Outcomes, labels and selection

Transaction records often contain provisional outcomes. A chargeback filed later can be reversed; a fraud investigation can close without confirmation; an AML alert disposition is not a criminal finding. Training labels should identify the authoritative outcome definition, horizon, maturation period and correction path. A reference table of case codes can help interpret the outcome, but it should not erase uncertainty. Model evaluation should distinguish confirmed outcomes, disputed cases, incomplete follow-up and decisions with no opportunity to observe the outcome.

Selection matters because the AI decision influences future data. A declined card authorization cannot produce the same chargeback observation as an approved purchase. A referred payment may be canceled by the customer. A rejected loan applicant will not generate repayment history for that loan. A model trained only on completed transactions observes a population shaped by prior policies. Review sampling, controlled experiments where appropriate and careful counterfactual analysis may help, but each requires governance and limits. It is incorrect to label every blocked transaction as fraud or every unobserved loan as good.

When the bank changes a threshold, the composition of settled transactions changes. A fall in observed fraud rate could mean better detection, more false holds or a shift in customer mix. Keep the decision and outcome tables linked so that evaluation can compare eligible, scored, held, released and confirmed cases. Reference data on segment or product should be point-in-time consistent across those populations. Otherwise a reporting join can create a false improvement.

A compact data contract

For the payment event, document instruction ID, source, payer and beneficiary keys, amount, currency, event timestamp, received timestamp, lifecycle status, status version, correction link and retention owner. For the customer and beneficiary reference, document source keys, canonical keys, relationship types, effective interval, recorded interval, confidence or approval where relevant, change reason and owner. For the derived feature, document eligible events, as-of join, aggregation window, freshness limit, null behavior, computation version and authorized uses. For the label, document source, outcome definition, horizon, maturity and revision handling.

Publish tests with the contract. An event ID must be unique within its source scope; allowed status transitions must be checked; active mappings must meet defined cardinality; joins must report unmatched and ambiguous rates; the feature must not use events after its decision cutoff. Set thresholds with business owners and record exceptions. A single aggregate pass percentage is insufficient when a small critical segment is entirely missing.

If a source team changes a field, it should notify downstream owners and provide a sample migration population. The feature and model teams should compare decisions on old and new interpretations before rollout. An emergency source correction may require a time-bounded fallback and impact review. These procedures make the contract a living control rather than a dictionary that becomes stale after the first release.

What to retain for replay

Retain references to the transaction event and its source version, the reference snapshot or effective records consulted, the entity-resolution result, the feature version and values or protected reproducible reference, the model output, policy version and final action. Protect sensitive identifiers and use role-based retrieval. A past decision should be explainable without querying mutable current tables as if they were historical truth. A corrected later view can sit alongside the original record with a clear provenance trail.

The bank can then answer a concrete question: did this payment receive a high risk score because of the customer's actual recent activity, a changed beneficiary mapping, a duplicated event, a stale feature or a later label accidentally joined into training? Distinguishing those causes is the practical value of treating transaction and reference data as different, versioned inputs to AI.

As a final check, have a business owner read three reconstructed records without seeing the implementation code. They should identify the event that triggered each decision, the customer and product relationship in force, the feature cutoff, the reason a reference record was selected and the action that followed. If the explanation depends on an undocumented query ordering or a current master-data value, the replay contract is incomplete. Preserve the disputed example as a test case after correcting the definition.

Correcting a beneficiary mapping

A payment at 14:00 refers to beneficiary alias B7. The reference service maps B7 to a customer relationship effective that morning, and the model computes a new-beneficiary feature. At 16:00 an investigator discovers that B7 was incorrectly merged with an unrelated payee. The source team corrects the mapping and records when the bank learned the truth. The 14:00 decision journal retains the mapping version actually used; a corrected analytical replay shows how the feature and score would differ.

The impact query finds every payment that joined through that mapping, then separates changed features, changed policy actions and actual customer effects. Reconciliation uses stable instruction IDs and final hub statuses. A current reference table alone cannot explain why a past score was produced. Conversely, preserving historic evidence does not excuse continuing to serve the wrong current mapping. This exercise makes transaction events and reference relationships distinct, versioned inputs to AI. Test a second alias belonging to the same actual payee and a joint account whose owners have different rights. A generic customer ID join must not collapse those relationships. Report unmatched and ambiguous mappings by affected decision count, not merely reference-table rows.