Experimentation & A/B Testing

Learning from controlled product experiments

A question and a comparison

An A/B test compares variants to estimate the effect of a planned change. Random allocation can make groups comparable when designed and implemented correctly. The method does not automatically make a banking change lawful, ethical or operationally safe.

State a falsifiable question, target population, unit of allocation, primary outcome and relevant harm measures. Improving comprehension of a payment status is a different objective from increasing credit acceptance. Define the minimum useful effect and the observation period before interpreting results.

Design an interpretable experiment

Randomise at an appropriate level: customer, account, business or another justified unit. Keep allocation stable where changing variants would confuse customers or contaminate measurement. Consider interference, such as colleagues sharing one business account, and avoid assuming each session is an independent person.

Plan sample size and duration using the expected baseline, effect and statistical approach. Validate allocation, instrumentation and unexpectedly imbalanced groups. Repeatedly inspecting results and stopping at the first favourable significance threshold can distort conclusions unless the method accounts for it.

Customer protection and release controls

Assess whether the change affects prices, terms, disclosures, eligibility, authentication or decisions. Necessary approval and review depend on the product and jurisdiction. Do not omit required information, protections or access to assistance simply to test whether less friction improves conversion.

Consider accessibility, vulnerable customers, privacy and appropriate customer information. There is no universal requirement for separate explicit consent to every interface experiment, nor a universal exemption because the change is called UX. Review the actual processing and customer effect.

Set stop conditions for material harm and technical failures, with an available decision-maker and rollback process. Rollback should stop new exposure and preserve evidence; existing financial effects may require reconciliation or remediation rather than a screen change.

Worked example: payment status wording

In this fictional test, two accurate status explanations are compared for payments awaiting confirmation. Customers are allocated consistently. The primary measure is correct understanding of the status; repeat submission attempts and support contacts are harm measures.

One variant increases customers' confidence but also increases mistaken belief that the recipient has already been credited. It should not win merely because fewer people open the help panel. The team revises the wording and confirms that backend status mapping remains accurate.

Interpret the results with limits

Report effect size and uncertainty, not only whether a threshold was crossed. Investigate meaningful subgroup differences using an appropriate analysis rather than selecting favourable segments after the result. Statistical significance does not establish commercial value or absence of harm.

Some outcomes take longer than the immediate journey. Lending losses, cancellations and complaints may mature after the experiment ends. Short observation periods should not be presented as proof of long-term profitability. Nonrandom pilots can support operational learning, but their causal conclusions need qualification.

From experiment to operation

Document allocation, versions, review decisions, results and limitations. A successful test still needs capacity and change-management checks before wider release. Monitor the rollout because the full population, partner workload or season may differ from the experimental setting.

The UK FCA Consumer Duty illustrates one scoped framework for assessing customer outcomes. It does not supply a universal experiment protocol; apply the actual market obligations alongside sound experimental design.

Takeaway

Experimentation should answer a useful question through a valid comparison within customer and operating controls. A conversion lift is evidence about one measure, not automatic permission to launch.

Continue to Digital Product Analytics.