Skip to main content

A/B Testing｜Creating an Experiment

The A/B Testing feature in Subotiz lets merchants compare different versions using real visitor traffic and experiment data. By configuring the experiment subject, audience, variants, traffic allocation, and performance metrics, merchants can validate changes before a full rollout, reduce business risk, and improve conversion performance and customer experience.
 

Preparing to Create an Experiment

Before creating an experiment, select the appropriate subject and complete the Subotiz A/B Testing SDK integration.
  1. Selecting an Experiment Subject: Subotiz supports two subjects for different experiment scenarios.
    • Pricing list: Use this subject to compare different Subotiz Pricing list configurations. After the SDK integration is complete, Subotiz automatically assigns visitors to experiment groups and displays the corresponding Pricing list without requiring additional page logic. This subject supports testing different pricing strategies, free trial offers, package combinations, and subscription plans.
    • Custom: Use this subject to test merchant-hosted pages, features, or business workflows. Subotiz assigns each visitor to an experiment group, while developers display the corresponding content or functionality based on the group assignment. This subject supports product pages, landing pages, registration flows, checkout flows, and new feature rollouts.
  2. Completing the SDK Integration: Both Pricing list and Custom experiments require the Subotiz A/B Testing SDK. After the initial integration is complete, merchants can reuse the same SDK setup for future experiments.
    • Initialize the SDK: Generate a stable experiment user ID, request the visitor’s experiment group, and automatically report experiment exposure.
    • Display the assigned version: For a Pricing list experiment, the SDK displays the assigned Pricing list. For a Custom experiment, developers display the corresponding page, content, or feature based on the group assignment.
    • Report experiment data: Use the SDK to report key user actions so Subotiz can calculate and compare results across experiment groups.
For integration steps, SDK parameters, and sample code, refer to the Subotiz A/B Testing SDK documentation.
 

Creating an Experiment

  1. Open the experiment page: Sign in to the Subotiz Admin, go to Data > A/B Testing, and click Create experiment.
 
  1. Enter the experiment name: Enter a clear name that makes the experiment easy to identify, such as:
    • Checkout Page Optimization
    • New Checkout Flow Experiment
    • Checkout UX Improvement
    The experiment name is only visible in the Subotiz Admin and is not shown to visitors.
 
  1. Select the subject: Choose Pricing list or Custom based on the content or experience being tested. Each subject uses a different implementation method, so confirm that the selection matches the intended development approach.
 
  1. Set the end conditions: Enter the target number of experiment users. The minimum target is 500 users. To limit how long the experiment can run, enable Maximum duration and enter a period from 1 to 365 days. Set the target based on actual traffic volume. A larger participant count generally provides more representative experiment results. The experiment ends automatically when it reaches either the target user count or the maximum duration.

 
  1. Select the audience: By default, all eligible visitors can enter the experiment. To run the experiment for a specific audience, select the targeted audience option.
 
  1. Configure the custom audience: After enabling audience targeting, add at least one filter rule.
    Subotiz currently supports the following conditions:
    • Country: Match visitors based on the country or region associated with their IP address. Search by country name or country code.
    • Traffic source: Match visitors based on the traffic source value passed through the SDK. The entered value must exactly match the value sent by the merchant’s application, such as Facebook.
    • Device: Run the experiment separately for desktop or mobile visitors.
    Multiple rules use AND logic. A visitor must meet every configured condition to enter the experiment.
 
  1. Enter the experiment description (optional): Describe the experiment objective and the main differences between the Control Group and Experiment Groups. A complete description provides additional context for AI-powered data insights and improves the accuracy of the resulting analysis.
 

Setting Up Variants and Traffic Allocation

Every experiment must include at least one Control Group and one Experiment Group. Subotiz automatically assigns eligible visitors to each group based on the configured traffic allocation.
  1. Name each experiment group: Enter a clear name for every group, such as:
    • Control Group
    • Variant A
    • Variant B
 
  1. Configure the group value: Complete the configuration based on the selected subject.
    • Pricing list: Select the Pricing list displayed to visitors in that group.
    • Custom: Enter a unique group variable that developers can use to identify the assigned group. Use lowercase letters, numbers, and underscores where possible.
Example:
Group name
Group variable
Control Group
control
Variant A
trial_7days
Variant B
no_trial
Confirm group variables with both the development and operations teams before implementation. The application must use the same values to avoid incorrect content assignment or reporting errors.
 
  1. Enter the group description (optional): Add a short description explaining what visitors in each group will see, such as:
    • Current Pricing list
    • Pricing list with a 7-day free trial
    • Updated CTA button copy
 
  1. Set the traffic allocation: Assign a percentage of eligible visitors to each group. The combined allocation across all groups must equal 100%.
    Example with two groups:
    • Control Group: 50%
    • Experiment Group A: 50%
    Example with three groups:
    • Control Group: 40%
    • Experiment Group A: 30%
    • Experiment Group B: 30%
 
  1. Add another variant: Click Add variant to test an additional version, then configure its name, group value, description, and traffic allocation.
 

Adding Custom Data

Only experiments with Custom selected as the subject support Custom data configuration. Pricing list experiments already include built-in metrics for views, checkout activity, payments, and revenue, so no additional data configuration is required.
Use custom data when testing behavior that Subotiz cannot automatically identify, such as registrations, button clicks, form submissions, or feature usage. Subotiz supports Direct data and Derived data.
 

Configuring Direct Data

Direct data represents raw events or values reported through the SDK, such as:
  • Completed registrations
  • Button clicks
  • Form submissions
  • Order amounts
Subotiz calculates the results for each experiment group using the selected aggregation method.
 
  1. Enter the display name: In Display name, enter the name shown in the Subotiz Admin, such as Registration Completed or Order Amount.
 
  1. Enter the data key: In Data key, enter the unique identifier used when reporting the event through the SDK, such as:
    1. signup_completed
    2. order_amount
    3. cta_click
The Data key must exactly match the value reported by the developer. Subotiz cannot collect or calculate the data when the values do not match.
 
  1. Select the aggregation method: In Aggregation, select the method that matches the reported data.
    • Count: Calculates the total number of times an event occurs, such as the total number of button clicks.
    • Unique users: Calculates the number of distinct users who trigger an event, such as users who complete registration.
    • Sum: Adds all reported numeric values, such as total order value.
    • Average: Calculates the average of all reported numeric values, such as average order value.
 
  1. Review common configurations: Use the following examples when configuring Direct data.
Display name
Data key
Aggregation
Checkout Started
checkout_started
Unique users
CTA Clicked
cta_click
Count
Order Amount
order_amount
Sum
Average Order Value
order_amount
Average
 

Configuring Derived Data

Derived data is calculated automatically from existing Direct data. Developers do not need to report an additional SDK event.
  1. Add Direct data: Configure at least one Direct data item before adding Derived data.
     
  2. Select the Derived data type: Subotiz currently supports the following types:
    • Ratio: Select existing Direct data for the numerator and denominator. For example, dividing Registration Completed by Page Viewed calculates the ratio between the two values.
    • Conversion rate: Select the Direct data used as the conversion event. Subotiz automatically uses the total number of users in the experiment group as the denominator. This supports metrics such as Checkout Start Rate and Payment Success Rate.
    • Retention rate: Select a retention period and the corresponding Direct data as the measurement basis. Retention rate represents the percentage of users who trigger a specified behavior for the first time and then trigger the same behavior again after the selected retention period. The system automatically calculates the retention rate based on the selected retention period and Direct data.
      Supported retention periods include:
      • Next-day retention (1 day)
      • 7-day retention
      • 30-day retention
      • Custom number of days
 
  1. Select metrics for the experiment objective: Configure data based on the behavior or outcome being tested.
Experiment objective
Recommended metrics
Page content optimization
Clicks, unique users who clicked, form completion rate
Registration flow optimization
Users completing each step, step conversion rate
New feature testing
Feature usage rate, average uses per user
Promotion testing
Coupon click rate, order amount, promotion page engagement
User engagement
Next-day retention, 7-day retention, 30-day retention
 
  1. Save the experiment as a draft: Click Save as draft after completing the experiment configuration.
    While the experiment remains in Draft status:
    • The experiment configuration can still be edited.
    • Subotiz does not assign visitors to experiment groups.
    • Subotiz does not collect experiment data.
    Review the subject, audience, variants, traffic allocation, group values, and data configuration before starting the experiment.
 

Best Practices

  • Confirm the implementation method: Select Pricing list only when testing Subotiz Pricing lists. Select Custom when developers need to control the page, content, or feature displayed to each group.
  • Use consistent group variables: Confirm every Custom group variable with the development team before launching the experiment.
  • Verify Data keys: Ensure every Data key exactly matches the value reported through the SDK.
  • Allocate all traffic: Confirm that the traffic allocation across all groups equals 100%.
  • Describe the experiment clearly: Record the experiment objective and the differences between groups to support future analysis and AI-powered insights.
  • Review the draft before launch: Confirm all settings before starting traffic allocation and data collection.
 
Creating an A/B Testing experiment requires selecting the appropriate subject, completing the SDK integration, and configuring the audience, variants, traffic allocation, and experiment data. After reviewing all settings, save the experiment as a draft and start it when the implementation is ready to collect real visitor data.
 

Appendix: Experiment Identifiers for Developers

The following identifiers support SDK implementation and experiment-data troubleshooting. Merchants generally do not need to configure them manually.
Identifier
Description
Experiment User ID
Generated automatically by the SDK based on the visitor’s device cookie. It keeps the same visitor in the same experiment group across visits.
Customer ID
The unique identifier Subotiz uses to recognize a customer on the platform.
External Customer ID
The customer identifier passed through payer_id when creating a Checkout Session. It connects the Subotiz customer record with the merchant’s own user record.