Data storage for AI workloads

Plain language meaning

Data storage for AI workloads explains how banks organise operational records, curated risk data, feature snapshots, training data, model inputs, logs and evidence stores so AI can work without breaking privacy, lineage, retention, access or audit obligations.

This topic is about bank-controlled storage for AI workloads. It is not about choosing a database brand, dumping all data into a lake or storing sensitive customer information without purpose.

In a real bank, this topic cannot be handled as a loose data-science or technology idea. It affects customer outcomes, fraud and AML control, operational queues, service continuity, privacy, security, model governance, audit replay, management reporting and regulatory confidence. AI should improve speed and quality, but the bank must still prove source data, permitted use, approved logic, human accountability, fallback handling and retained evidence.

Where it sits in the banking AI journey

This card belongs to AI Data and Model Operations. The working flow is Source records, Curated storage, Feature snapshots, Model and audit stores, and Controlled consumption.

Read the flow as a bank operating model. Each stage needs a source system, a data owner, a timing rule, a quality gate, a model or rule boundary, an exception path, a customer-impact view, a fallback option, a monitoring requirement and a retained record. That is what separates useful AI adoption from uncontrolled automation.

Banking data and evidence

The important data points are customer master, account record, transaction history, feature snapshot, training label, model input, audit log, and retention marker. These items matter because they can influence risk scoring, operational repair, fraud action, AML triage, customer treatment, reporting, model monitoring and management decisions.

The evidence pack should include data catalogue, access review, encryption record, retention rule, lineage entry, quality report, and storage usage report. A strong bank can replay the journey from source data to transformed input, AI output, rule result, human action, system outcome and monitoring result. A weak bank only knows that a process ran and hopes the process was right.

Controls that make AI adoption safe

The core controls are data classification, access entitlement, encryption, retention policy, lineage tagging, quality gates, and storage cost control. These controls keep the topic anchored to banking purpose, approved policy, data governance, model-risk expectations, operational resilience, customer fairness, privacy, security and auditability.

The practical design should define what AI may recommend, what it must never decide alone, which deterministic rule remains authoritative, who owns thresholds and overrides, how degraded service is handled, how customer harm is detected and what evidence is retained. Without that control design, faster AI can simply make weak processes fail faster.

Architecture and data-operation lens

Banking AI depends on the architecture around it. Storage, streams, feature definitions, training sets, model versions, thresholds, feedback labels and rollback paths must be governed before the bank relies on AI output. The model is only one part of the control chain.

A bank-grade design connects channels, source systems, core records, payment hubs where relevant, fraud systems, AML platforms, case tools, data platforms, feature stores, model-serving endpoints, policy engines, audit logs and management dashboards. It also records degraded operation, recovery actions and lessons learned.

Regulatory and governance lens

Federal Reserve SR 26-2, dated 17 April 2026, gives revised model-risk guidance for traditional models and non-generative AI models used by banking organisations, including development, validation, monitoring, change control and governance.

The Federal Reserve's 2026 model-risk guidance states that generative and agentic AI are outside that guidance, while broader bank risk-management and governance practices still need to control tools and processes not covered by the guidance.

NIST AI RMF 1.0 uses Govern, Map, Measure and Manage functions for AI risk management, and NIST AI 600-1 adds generative-AI risk actions for source grounding, content provenance, data protection, cybersecurity and human oversight.

FFIEC Architecture, Infrastructure and Operations guidance expects financial-institution technology environments to be governed, resilient, secure, monitored, documented and aligned to business risk, including emerging technologies such as artificial intelligence and machine learning.

BCBS 239 remains current for effective risk data aggregation and risk reporting, and the Basel Committee's January 2026 newsletter reiterates the importance of accurate, comprehensive and timely data capabilities in banks.

FFIEC BSA/AML examination guidance expects suspicious activity monitoring systems and supporting technology to be risk-based, explainable by management, independently tested where appropriate and aligned to the bank's risk profile.

FinCEN's 12 June 2026 Section 314(b) materials clarify information sharing for possible terrorist activity, money laundering and fraud-related specified unlawful activity within the statutory safe-harbor framework for participating financial institutions.

OFAC's Framework for Compliance Commitments describes sanctions compliance programme components including management commitment, risk assessment, internal controls, testing and auditing, and training.

Diagram walkthrough

Read the diagram from left to right as Source records, Curated storage, Feature snapshots, Model and audit stores, and Controlled consumption. It is a banking control map. The point is to show how data, AI or ML output, rules, human action, operational routing and audit evidence should connect.

Use it as a 30-minute study method. For each box, ask which system creates the data, which definition is used, which model or rule acts, what can go wrong, who can override it, how a fallback works, which customer or regulatory impact exists and what record proves the final state.

Most important mistake to avoid

The common failure is building an AI lake that is technically rich but bank-poor: unclear ownership, weak classification, excessive personal data, unreliable lineage and no proof that a model used the right version of the data.

The correction is disciplined scope. Keep the chapter anchored to banking purpose, prove the data path, make ownership visible, test failure behaviour, record the evidence and make the final outcome explainable without relying on memory, assumptions or developer-only knowledge.

A store designed around a dated decision

A bank may keep raw payment events, curated analytical tables, model features and decision logs in different stores. Each has a different purpose. The raw event preserves what arrived and when. A curated table applies agreed identity and status definitions. A feature snapshot contains values permitted at a decision cut-off. A decision log connects the versioned score to policy and final action. Copying all fields into one unrestricted repository is neither necessary nor a sound access design.

Imagine an investigator asks why a payment was referred on Monday. The system must recover the event and reference-data versions available then, not today's corrected beneficiary record. Storage design therefore includes stable identifiers, event and ingestion timestamps, schema versions, retention policy and controlled links among layers. A late correction should be added as a correction with provenance rather than silently replacing the original evidence. The bank should document which layer is authoritative for each question.

A useful acceptance test retrieves one decision end to end and checks that an authorised reviewer can reproduce it without granting broad access to unrelated customers. Then test a deleted or expired source record, a duplicate event and a schema change. The team should know when a replay is impossible and say so explicitly. Encryption and access controls protect stored data, but they do not establish that the feature was valid at the decision point. The NIST AI RMF Measure playbook is a general reference for documenting data provenance and dependencies; bank-specific retention and privacy duties require separate legal and operational assessment.

Store for the decision and the evidence

Banking AI needs several forms of data storage because its jobs differ. A fraud service needs low-latency feature lookup at payment time. A training job needs reproducible historical snapshots and later outcomes. A case investigator needs authorized source evidence. An internal policy assistant needs approved document versions and a retrieval index. An audit reviewer needs the original decision context after the live database has changed. Placing everything in one store obscures different freshness, access, retention and consistency requirements.

Begin with the data lifecycle. Identify the authoritative source, ingestion path, curated representation, feature computation, model request, policy action, outcome and retention endpoint. Define the unit and clock at each step. A payment hub may be authoritative for instruction and settlement status, while a feature store holds derived velocity counts. The feature store is not a replacement for the payment ledger. A vector index is a search aid, not the authoritative policy document repository.

Design storage around a concrete recovery question: can the bank reproduce what the model saw and what action followed at a particular time? A mutable current-state table alone cannot answer that. Use versioned records, protected decision snapshots or reproducible event history as appropriate. The exact combination depends on scale, legal retention and access requirements.

Operational and analytical stores

Operational systems maintain current transactions, accounts, applications and cases under strict consistency and business controls. AI should consume governed interfaces or copies rather than running heavy training queries against critical transaction systems. Analytical stores can hold curated history for feature development, monitoring and reporting. Reconcile the copy to source control totals and document its lag and exclusions. A successful ETL job does not prove that every eligible payment arrived.

Warehouse or lakehouse tables support large joins and reproducible batch runs. Partition by useful time and product keys, but avoid partitioning that hides late corrections or creates skew. Keep schema and transformation versions, input manifests and quality results. A credit model training set should identify its application cohort, observation cutoff, label horizon and reference snapshots. A table named "latest" is not sufficient evidence of what was used in a prior validation.

An online store serves features under a latency budget. It may keep customer velocity counters, account age or recent behavior, with timestamps and validity states. A cache can improve response time but can also serve stale or partial values. Define time-to-live, update semantics, recovery from snapshot and fallback behavior. A valid zero and an unavailable count must be distinguishable. Measure both lookup latency and feature freshness at the decision boundary.

Event logs and change history

An append-oriented event log can preserve payment and case lifecycles, supporting stream processing and replay. Its events need stable business IDs, source sequence, event type, event time, recorded time and schema version. Technical retries should be linked to the same instruction. A payment acceptance, settlement and return are distinct events; treating them as duplicate rows or independent payments corrupts features and audit evidence.

Change-data capture from a database can deliver updates efficiently, but a row change may lack the business meaning of an event. A deleted record or corrected customer link needs explicit handling. Snapshot plus change log requires a consistent handoff watermark. If a consumer restores a snapshot and replays from the wrong offset, it can lose or duplicate transactions. Test this under realistic volumes and partitions.

Retention must support the intended reconstruction period without uncontrolled preservation of sensitive raw data. Some evidence may be stored as protected references and versioned transformations; other use cases need a decision-time feature snapshot. Test whether old records can be read after schema migration and archival. A pointer into a table that is routinely overwritten is not an audit trail.

Feature stores

A feature store can centralize definitions and serve consistent values to training and production. Its value depends on semantic contracts, not the label on the product. A feature definition should include entity, eligible events, time window, transformation, source lineage, freshness, null handling and owner. The offline path should reconstruct values as available at historical decision time. The online path should return the corresponding current values with status.

Compare online and offline values on sampled request IDs. A batch calculation may include late events that the online service never saw, so the comparison must respect availability time. Investigate discrepancies by segment and event boundary. A model trained on a polished historical feature can underperform if the live value is often absent or stale. Record the feature version with each model artifact and decision.

Do not let a store silently default missing values. If a streaming partition is down, returning zero payment velocity may make customers appear low risk. The service should report observation timestamp, source watermark and validity. The model or policy can then follow an approved missing-data path. Cache failure, source failure and genuine absence of activity should have different statuses.

Document and vector storage

A retrieval-augmented assistant may use a document repository, extraction pipeline, chunk store and vector index. The authoritative document needs owner, approval status, effective date, version and access scope. Chunks and embeddings inherit those attributes and need links to source spans. A vector similarity result is a candidate passage, not proof of authority or correctness. Retrieval should filter by permissions and validity before the model drafts an answer.

Index updates can lag document approval or withdrawal. Define a publication process that prevents obsolete policies from continuing to answer current questions. Test a superseded document, a future-effective rule and a topic without an approved answer. Record index version and retrieved passage IDs for each response. A later rebuild should not erase the evidence of what an earlier assistant saw.

Embeddings are version dependent. Changing the encoder, chunking strategy or normalization can alter neighbors while the source documents stay the same. Reindex under a new version, evaluate retrieval and citation quality, and control rollout. Do not mix incompatible vectors in a single index without a deliberate design. The final human-reviewed communication should be stored separately from the generated draft.

Security boundaries

Storage choices determine who can see raw account activity, bureau data, case narratives and customer identity links. Grant minimum access for training, serving, operations and investigations. A feature service may return a derived count without exposing full transaction history. A validator may use controlled samples. An authorized investigator can retrieve source detail for a specific case. Log access and exports, and review service identities and vendor permissions.

Encryption, key management, backups and deletion need to cover online, analytical and archive copies. A training export on a shared file store can undermine tight permissions on the source database. A vector index containing chunks of sensitive documents is itself sensitive. Pseudonyms may remain linkable and should not be treated as anonymous merely because names are removed.

Retention periods and correction procedures differ by data class and jurisdiction. Work with legal, privacy and records owners to define them. Preserve what is needed to explain and remediate decisions under applicable obligations, while avoiding indefinite storage of prompts and raw documents by default. A source correction may require updating current features and keeping a marked original decision record under controlled access.

Consistency and transaction boundaries

The AI decision path crosses multiple systems. A model may score a payment, a policy service may hold it, and a case tool may open a review. These operations can succeed or fail separately. Use stable IDs and reconciliation to identify partial states. An audit record that says "hold" when the payment hub actually settled is misleading. The final action must be tied to the authoritative transaction outcome.

Strong consistency is important for some state, such as preventing duplicate payment execution. Feature aggregates may be eventually consistent within an approved freshness limit. State the acceptable lag and what happens when it is exceeded. A ledger posting and an analytical feature update need not be simultaneous, but the model should know which state it is using. A cache should not present an old value as current.

Idempotency protects against retries across storage layers. A message replay should not increment a count twice, open a second case or execute another transfer. Keep business instruction IDs distinct from message attempt IDs. Test recovery from consumer restarts, partial commits and network timeouts. The phrase "exactly once" in a database component does not guarantee exactly one business action across the whole workflow.

Capacity and performance

Training jobs scan large histories; online scoring needs predictable tail latency. Separate resources or schedule workloads to avoid a training query slowing critical decisions. Size online stores for peak request rates, hot keys and recovery. A single large corporate customer can concentrate activity on one partition. Monitor memory, disk, compaction, queue lag and the age of the oldest feature. Average lookup time can conceal a few high-risk requests timing out.

Batch storage needs cost discipline. Keep reproducible manifests and required evidence, but review unused copies and redundant extracts. Partition and index for common investigations: decision period, model version, source version and affected customer or instruction under authorized access. Test a reverse-lineage query that finds every decision using a defective mapping. If it takes weeks of manual work, the archive is not operationally useful.

Backups should be restorable, not merely present. Test a restore with schema versions and access controls intact. A feature store rebuilt from an event log needs an initial snapshot and complete replay offsets. During rebuild, use an approved limited mode rather than serving empty counters as valid zeros. Validate a sample of reconstructed features before returning to normal decisions.

Payment architecture example

A hub emits instruction and status events into a durable stream. A processor deduplicates by business ID, updates a recent-payment counter and publishes an online feature with watermark. The scoring service reads the feature; a policy engine decides whether to release or hold. A decision journal stores protected references to source event, feature version, model output, policy version and final hub action. Analytical tables later join returns and confirmed fraud labels for evaluation.

If the processor lags, the feature lookup should signal stale data. The policy invokes the approved fallback for affected payments. When the stream recovers, replay repairs feature state, but it does not re-execute past instructions. Reverse lineage identifies payments that were decided with stale or missing values. An investigator can compare original and corrected features while preserving the actual historical action.

Credit architecture example

An application system provides submissions and revisions. Bureau and account data arrive with observation and received dates. A curated analytical layer builds point-in-time features for model development; a controlled serving path supplies the same definitions for live applications. The model registry stores the approved artifact and training dataset reference. A decision journal connects score, affordability rules, underwriter review and final customer outcome.

A bureau cache correction may change current data. The bank retains the original response used by a past application under appropriate controls and calculates a corrected scenario. It identifies affected applicants and distinguishes changed scores from changed decisions. A storage design that retains only the latest bureau row makes that review unreliable.

Acceptance exercise

Select a payment, credit application and policy-assistant answer. Identify the authoritative source for each fact, the derived stores, version and timestamp at decision time, access route for an authorized reviewer, and retention or correction path. Simulate a streaming outage, a source correction and a withdrawn policy document. Verify that the live workflow follows its approved mode, that earlier decisions remain explainable and that restored data does not create duplicate actions.

The best storage design is the one that lets the bank serve the right input on time, restrict access appropriately, reproduce a past decision and repair a defect with a known affected population. A fast database or fashionable architecture cannot substitute for those properties.

Choose storage by workload

For a transaction ledger, the primary concern is authoritative balances, postings and reconciliation. Model experiments should not make it serve arbitrary large joins. For a feature cache, predictable reads and fresh values matter; it may be rebuilt from governed sources but should not be treated as the ledger. For a historical analytical table, reproducibility and scalable scans matter more than millisecond reads. For a decision journal, integrity, retention and traceability matter. Write these requirements down before choosing a product or cloud service.

An in-memory online store can provide low latency but is vulnerable to lost state or stale cache. Define whether it is a cache of a durable source or the sole holder of a derived aggregate. If it is rebuilt, specify snapshot and log handoff, warm-up criteria and fallback during recovery. A relational store may offer strong transaction semantics for case state but can be strained by high-frequency feature writes. An object store can hold versioned training data economically but needs catalog, schema and access governance. Each is a component of a decision architecture, not an interchangeable bucket.

Keep the authoritative source identified in metadata. A feature store might expose account age calculated from an account master; it does not own the opening-date correction process. A vector index might surface a paragraph of policy; it does not approve the policy. A reporting mart may show a fraud disposition; the case system governs its workflow. If copies disagree, the contract should say which source is authoritative and how reconciliation works.

Partitioning and keys

Partition large analytical data by dates that reflect query and retention needs, while preserving event and recorded times. An event-time partition can receive late arrivals; the pipeline should handle them without silently dropping an old partition. A processing-time partition helps track what the bank knew by a cutoff. Both may be needed for point-in-time model evaluation. A partition key alone does not define the decision timestamp.

Use stable business IDs with source scope. Two acquired banks may reuse local account numbers. A payment can carry an instruction ID, message ID and settlement ID. Store typed relationships between them. An online feature keyed only by a local account number could mix unrelated customers; a batch join could multiply rows. Test key collisions, missing mappings and changes after mergers. Where tokenization is used, manage rotation and preserve controlled linkage for authorized replay.

Hot-key skew can undermine a low-latency store. A large merchant, correspondent or corporate customer may receive far more events than ordinary accounts. Partitioning by entity keeps local order but concentrates load. Sharding and aggregation can help, yet change the consistency model. Stress test high-volume entities and reconciliation after recovery. Average throughput is not a sufficient capacity measure.

Schema evolution

Additive fields may be easy to ingest, but meaning changes require deliberate versioning. If "payment status 3" changes from accepted to settled, historic and current records cannot share an unqualified mapping. Preserve schema versions and business dictionaries. A model feature based on status needs revalidation. Test old archived records with the current reader and keep a migration map where necessary.

Changing numeric units is particularly dangerous. An amount stored in minor currency units can be interpreted as major units; a percentage can be represented as 0.05 or 5. An online store may accept either as a number. Include units and allowed ranges in contracts and test representative currencies and products. A feature or threshold can be wrong by orders of magnitude while serialization succeeds.

Deletes and corrections need semantics. If a transaction is reversed, retain its lifecycle rather than deleting it from history as if it never happened. If a customer record is rectified, update current uses and keep required prior-decision evidence under controls. If a document is withdrawn, remove it from live retrieval while preserving appropriate evidence of earlier answers. Each store should have a procedure for propagation and verification.

Point-in-time analytical snapshots

A training set should not be a live query whose result changes with every source correction. Freeze an extract or create a versioned manifest of input partitions, reference records, feature code and label rules. Record when source data was available, not only when the modeled event occurred. Validate that labels follow the decision and mature over the chosen horizon. Retain quality reports and exclusions.

Time travel in a storage engine can help, but it is not a complete point-in-time guarantee. A snapshot of a table as it existed yesterday may already contain information that arrived after a decision six months ago. The pipeline must use historical availability timestamps and effective reference intervals. Test sample applications or payments against original decision records. If the source never stored recorded time, disclose the limitation in model validation.

Dataset versioning should cover transformations and cohort definitions. Two datasets with the same source snapshot but different filters are different training evidence. Record code revision, parameters, random seeds where relevant and model training configuration. A validation report should identify the exact dataset version. If a later source correction alters outcomes, rerun material analyses under a new version rather than silently editing old results.

Decision journal design

The journal should record one business decision with links to technical attempts. Fields can include decision ID, source instruction or application ID, event and decision timestamps, feature snapshot reference and validity, model request and artifact, score or abstention, policy version, fallback reason, human override and final action reference. Each field should have a defined meaning and retention class. Sensitive raw values may live behind controlled references rather than in every log span.

Append status transitions or keep versioned records. A payment hold followed by analyst release and eventual return should not collapse into one current status. A credit application can be revised and reassessed. A generative assistant draft can be edited before sending. The journal should distinguish what the AI proposed from what the bank actually did.

Reconcile journals to authoritative systems. Count eligible payment instructions, scored and fallback paths, case actions and ledger statuses by window. Define expected exclusions and retries. An immutable log that omits failures or cannot link to final outcomes gives false assurance. Monitor journal ingestion itself and establish a degraded mode if evidence capture fails.

Disaster recovery

Backup and recovery objectives depend on the use. A training warehouse can be rebuilt over hours or days from governed sources if manifests and source retention permit it. A payment feature service may need rapid recovery or a well-tested rule-only path. A decision journal may need durable local buffering if its central archive is unavailable. Specify recovery point and time objectives in terms of lost or delayed decisions, not just server restoration.

Run a recovery drill that restores an online store from snapshot plus event log, validates a sample of feature values and gradually resumes scoring. Include a duplicated event and a missing partition. Test a batch dataset restore with old schema versions and a decision journal query for an archived case. Verify permissions after restore; an emergency backup should not become an unrestricted copy of customer data.

Correlated failures require attention. If the same cloud region hosts model serving, feature store, case queue and evidence sink, a failover claim may be illusory. An independent manual path needs staff, access and source information. Test under peak load and partial network failure. A hot standby with stale data can be as problematic as an unavailable service.

Data lifecycle and minimization

Classify raw source records, curated tables, derived features, training extracts, model artifacts, prompts, retrieved passages and decision logs separately. Each has a purpose and audience. Set retention and deletion rules with legal and privacy owners, including backups and vendor copies. A derived value can still be personal data when linked to an account. Restrict broad copying and record exports.

When a data subject or source owner corrects a record, identify affected current features and past decisions. Current serving state may need immediate recomputation; a training dataset may need a future revised version; a prior decision journal may need a linked correction rather than destructive overwrite. The bank should be able to explain the original action and the later remediation while following applicable obligations.

For a retrieval index, remove superseded or unauthorized chunks from live answers and validate that filters work. Preserve the source-version reference needed to investigate earlier output under the retention policy. A model's generated text may include customer information that did not belong in telemetry. Audit both content and access paths.

Cost and reliability review

Estimate storage growth from event frequency, feature snapshots, model attempts and outcome retention. Do not collect every intermediate tensor or raw prompt by default. Retain enough to reproduce and challenge decisions, with sampling or protected references for less critical diagnostics. Test retrieval of evidence within incident timelines. A cheap archive that takes days to access may impede remediation.

Measure cost alongside decision reliability. Caching can reduce latency but introduces staleness; compacting an event log can save space but remove required history; aggressive deduplication can merge legitimate payment events. Every optimization should state its effect on replay, privacy and fallback. Review these tradeoffs with business and control owners, not only infrastructure engineers.

Storage review questions

Which store is authoritative for the payment event, customer identity, feature, model output and final action? Can an old version still be interpreted after schema change? Does online lookup return age and validity? Can a training snapshot be reproduced without future leakage? Does a reverse-lineage query enumerate decisions affected by a source defect? Can sensitive data be retrieved only by authorized people? Has a full restore been tested?

Use sample decisions and fault drills to answer. If the answers depend on a current table or an engineer's memory, the storage design needs a stronger evidence contract before the AI workload is considered reliable.

A practical sign-off record

Write down one real payment ID, its source event version and the online feature vector actually returned. Record the feature store's watermark, model artifact and response, policy action, case link and authoritative final payment status. Ask a second reviewer to retrieve these from the storage tiers under ordinary authorized access. Time the retrieval and note any missing version or ambiguous join.

Repeat with a batch-scored credit application from an archived run. The reviewer should find its cohort inclusion rule, source and reference snapshots, calculated features, model and policy versions, publication gate and later outcome. Then introduce a corrected bureau record and enumerate affected decisions without changing the original run manifest. If the bank can only recover the latest application row, it cannot fully review the old decision.

Finally, sample a policy-assistant answer after its cited document has been superseded. The reviewer should see the source version and effective date supplied to the model, the draft, the human edit and final communication. Test that the obsolete document no longer appears in live retrieval and that archived evidence remains access controlled. These three records expose different storage weaknesses and give the sign-off a concrete basis.

Restore a feature store without false zeros

An online payment-velocity store loses a partition. The processor restores a snapshot at a known offset and replays events from a durable log. Until the watermark and sampled counters reconcile to source business IDs, its values remain invalid for affected customers. The policy uses an approved limited mode. A cache that answers zero during warm-up can make active customers appear low risk.

After restore, compare one held and one released payment's stored decision-time feature with independent source reconstruction. The current repaired value is not the historic input. Keep protected references to the original score and final payment status in the decision journal, with access restricted to authorized reviewers. Test the same process after schema migration and backup recovery. Storage is fit for AI only when serving, replay and evidence all work under failure. Test a backup copy with the same role permissions and retention as production. An emergency restore should not make raw payment narratives broadly visible. Verify that archived schema versions can still be read and that a corrected source record links to, rather than overwrites, the historic decision.