The A/B Testing feature in Subotiz lets merchants compare multiple versions of the same customer experience using real visitor behavior and business data. By measuring conversion rates, revenue, and other experiment metrics, merchants can determine which version performs better before making it available to all visitors. This helps merchants make data-driven decisions, reduce rollout risk, and continuously improve customer experience and business performance.
Understanding the A/B Testing Process
After an experiment is activated, Subotiz automatically assigns eligible visitors to experiment groups and tracks the performance of each version. Merchants only need to configure the experiment and prepare the corresponding content, Pricing list, pages, or features for each experiment group.
- Receive a visitor: A visitor enters the merchant's website or applicable business flow.
- Assign an experiment group: Subotiz automatically assigns the visitor to either the Control Group or an Experiment Group. The same visitor remains in the assigned group throughout the experiment.
- Display the assigned version: Each group sees a different version. For example, the Control Group may continue to display the current Pricing list, while the Experiment Group displays an alternative version.
- Collect experiment data: Subotiz continuously tracks conversion rates, revenue, and other configured metrics for each group.
- Evaluate the results: Once the experiment reaches a sufficient sample size and a high confidence level, merchants can determine whether to make the better-performing version available to all visitors.
Key Concepts
The following terms appear throughout the A/B Testing feature. Understanding these concepts makes it easier to create experiments and interpret the results.
- Control Group: The version that remains unchanged throughout the experiment. It typically represents the current live experience and serves as the benchmark for comparison.
- Experiment Group: A variation tested against the Control Group. A single experiment can include multiple Experiment Groups, such as Experiment Group A and Experiment Group B. Each group is measured independently and compared with the Control Group.
- Traffic Allocation: The percentage of eligible visitors assigned to each group. The combined allocation across all groups must equal 100%. For example, allocating 50% of traffic to the Control Group and 50% to Experiment Group A distributes eligible visitors evenly between the two versions.
- Confidence Level: A statistical measure that indicates how likely an observed difference between an Experiment Group and the Control Group reflects a real performance difference rather than random variation. A confidence level of 95% or higher is generally considered statistically reliable.
Common Use Cases
A/B Testing can be used in almost any business scenario where merchants want to validate changes using real customer data.
- Pricing and Subscription Strategies: Compare different Pricing list configurations, free trial offers, package combinations, or subscription plans to measure their impact on conversion rates and revenue.
- Page Content and Copy: Compare product titles, page copy, button labels, promotional messages, or other content to determine which version generates higher engagement and conversion rates.
- Checkout Experience: Compare different checkout page layouts or templates to identify the version that achieves a higher checkout completion rate and lower abandonment.
- User Onboarding Flows: Compare different registration or onboarding flows to identify where users leave the process and which experience delivers a higher completion rate.
- New Feature Rollouts: Release a new feature to a limited percentage of visitors and measure adoption, engagement, and usage before making it available to all users.
When to Use A/B Testing
A/B Testing is especially useful when:
- You're unsure whether a proposed change will improve or reduce conversion performance.
- You want to validate a change with a limited audience before a full rollout.
- You need measurable data to support business decisions instead of relying on assumptions or intuition.
- You want to evaluate customer adoption before releasing a new page, feature, or experience to all users.
Business Value of A/B Testing
- Measure performance: Quantify the impact of each change on conversion rates, revenue, and other key business metrics.
- Reduce rollout risk: Validate changes with a limited audience before making them available to all visitors.
- Make evidence-based decisions: Compare variations using measurable performance data rather than assumptions or opinions.
- Continuously optimize: Use experiment results, performance analysis, and AI-powered insights to improve future decisions and customer experiences.
Best Practices
- Test one variable at a time: Change only one primary element, such as the Pricing list, button copy, or page layout, so the source of any performance difference remains clear.
- Collect enough data: Allow the experiment to reach an adequate sample size and confidence level before making a decision.
- Avoid ending experiments early: Large percentage changes based on limited traffic may not be statistically reliable.
- Ensure sufficient traffic: If visitor volume is low, extend the experiment to collect a more representative sample.
- Review the overall business impact: Consider revenue, conversion performance, customer experience, operational effort, and potential risk before rolling out a winning variation.
A/B Testing helps merchants validate changes using real customer behavior and measurable business data before a full rollout. By selecting the right experiments, collecting sufficient data, and evaluating results objectively, merchants can reduce rollout risk while continuously improving conversion performance, revenue, and customer experience.