Documentation

Everything you need to know about setting up and using ABTestCraft for A/B testing in Craft CMS.

Installation

From the Plugin Store (Recommended)

  1. In the Craft Control Panel, navigate to Plugin Store
  2. Search for "ABTestCraft"
  3. Click Install

Via Composer

bash
composer require livehand/craft-abtestcraft

After installation, navigate to Settings → Plugins in the Craft Control Panel and install the plugin.

Requirements

  • Craft CMS 5.0 or later
  • PHP 8.2 or later

Quick Start

Follow these steps to create your first A/B test:

1. Create Your Test Entry

Navigate to the section where you want to run a test. Create or select an entry to be your control (the original version).

2. Create a Variant

Duplicate the control entry to create a variant. Make your changes—new headline, different layout, alternate CTA, etc.

3. Configure the Test

In the Control Panel, go to ABTestCraft → Tests and click New Test. Select your control entry and variant entry.

4. Set Up Goals

Define what counts as a conversion. Choose from:

  • Form submissions
  • Button/link clicks
  • Page views
  • File downloads
  • Custom events

5. Start Testing

Activate your test. Visitors will be randomly assigned to either the control or variant, and their conversions will be tracked.

Goal Types

ABTestCraft supports multiple conversion goal types to match your business objectives.

Phone Clicks

Track clicks on phone number links (tel: links). Useful for local businesses and service providers.

html
<a href="tel:+1234567890">Call Us</a>

Email Clicks

Track clicks on email links (mailto: links).

html
<a href="mailto:[email protected]">Email Us</a>

Form Submissions

Automatically track form submissions from Freeform or Formie. No additional configuration required.

File Downloads

Track when users download specific files:

html
<a href="/files/brochure.pdf" data-ab-track="download">Download Brochure</a>

Page Views

Track when users visit a specific page (e.g., a thank you page):

twig
{% if craft.abtestcraft.hasActiveTest() %}
  {{ craft.abtestcraft.trackPageView('thank-you') }}
{% endif %}

Custom Events

Track any custom interaction using JavaScript:

javascript
window.ABTestCraft.trackConversion('video_watched');
window.ABTestCraft.trackConversion('add_to_cart', { productId: '123' });

Configuration

Configure ABTestCraft through the Control Panel or a config file.

Control Panel Settings

Navigate to Settings → ABTestCraft to configure:

  • Cookie Duration: How long variant assignments persist (default: 30 days)
  • Significance Threshold: Statistical confidence level required (default: 95%)
  • Minimum Detectable Effect: Smallest effect size to detect (default: 10%)
  • Conversion Rate Limit: Max conversions per visitor per minute (default: 10)
  • GA4/GTM Integration: Enable/disable dataLayer events

Auto-Tracking Options

Toggle automatic tracking for:

  • Phone clicks (tel: links)
  • Email clicks (mailto: links)
  • Form submissions (Freeform/Formie)
  • File downloads

Conversion Counting Modes

  • First Only: One conversion per visitor total
  • Per Goal Type: One conversion per goal type per visitor (recommended)
  • Unlimited: Record all conversions

Config File

Create config/abtestcraft.php:

php
<?php

return [
    // Core settings
    'cookieDuration' => 30, // days
    'significanceThreshold' => 0.95,
    'minimumDetectableEffect' => 0.10,
    'conversionRateLimit' => 10,
    'conversionCountingMode' => 'per_goal_type',

    // GA4/GTM integration
    'enableDataLayer' => true,

    // Auto-tracking toggles
    'trackPhoneClicks' => true,
    'trackEmailClicks' => true,
    'trackFormSubmissions' => true,
    'trackFileDownloads' => true,

    // Notifications
    'sendSignificanceEmail' => true,
    'notificationEmails' => '[email protected]',
];

Environment Variables

Override settings per environment:

php
'cookieDuration' => getenv('ABTEST_COOKIE_DURATION') ?: 30,
'enableDataLayer' => getenv('ABTEST_ENABLE_DATALAYER') === 'true',

Twig Variables

Access test data in your templates using the craft.abtestcraft variable.

Check for Active Test

twig
{% if craft.abtestcraft.hasActiveTest() %}
  {# User is participating in a test #}
{% endif %}

Get Active Variant

twig
{# Returns 'control' or 'variant' string #}
{% set variant = craft.abtestcraft.getActiveVariant() %}

Get Test Information

twig
{% set test = craft.abtestcraft.getActiveTest() %}
{% if test %}
  <p>Test: {{ test.name }}</p>
  <p>Started: {{ test.startDate|date }}</p>
{% endif %}

Boolean Variant Checks

twig
{# Quick boolean checks #}
{% if craft.abtestcraft.isShowingVariant() %}
  {# User is seeing the variant #}
{% endif %}

{% if craft.abtestcraft.isShowingControl() %}
  {# User is seeing the control #}
{% endif %}

{# Check specific variant by name #}
{% if craft.abtestcraft.isVariant('variant-a') %}
  {# Show variant A content #}
{% endif %}

Track Page View

twig
{{ craft.abtestcraft.trackPageView('checkout-complete') }}

Check if Entry is a Variant

twig
{% if craft.abtestcraft.isVariantEntry(entry) %}
  {# This entry is being used as a variant in a running test #}
{% endif %}

Form Plugin Helpers

twig
{# Check if a form plugin is installed #}
{% if craft.abtestcraft.hasFormPlugin() %}
  {# Freeform or Formie is available #}
{% endif %}

{# Get list of installed form plugins #}
{% set plugins = craft.abtestcraft.getInstalledFormPlugins() %}

{# Get all available forms #}
{% set forms = craft.abtestcraft.getAvailableForms() %}

Cascade Support

ABTestCraft supports hierarchical testing where child pages automatically inherit their parent's test assignment.

How Cascade Works

When a visitor is assigned to a variant on a parent page, all child pages under that parent will also show variant content. This is useful for:

  • Testing entire sections of a site
  • Ensuring consistent experience across related pages
  • Running tests on landing pages that affect child content

Template Functions

twig
{# Check if current page is part of a cascaded test #}
{% if craft.abtestcraft.isCascaded() %}
  {# This page inherits a test from a parent #}
{% endif %}

{# Get parent accounting for cascade #}
{% set parent = craft.abtestcraft.getParent(entry) %}

{# Get children accounting for variant borrowing #}
{% set children = craft.abtestcraft.getChildren(entry) %}

How Variant Borrowing Works

When a parent entry is in a test:

  1. Control visitors see the control parent and its original children
  2. Variant visitors see the variant parent and its children (which may be different)

The plugin automatically maps descendants and updates these mappings when entries are moved or restructured.

Performance Note

For entries with 50+ descendants, cascade mappings are processed asynchronously to avoid blocking page loads.

JavaScript API

Use the JavaScript API for client-side conversion tracking.

Track Conversions

javascript
// Simple conversion
window.ABTestCraft.trackConversion('button_clicked');

// With metadata
window.ABTestCraft.trackConversion('purchase_complete', {
  value: 99.99,
  currency: 'USD',
  orderId: '12345'
});

Check Active Test

javascript
if (window.ABTestCraft.hasActiveTest()) {
  console.log('User is in a test');
}

Get Variant Info

javascript
const variant = window.ABTestCraft.getVariant();
console.log(variant.id, variant.name);

Listen for Events

javascript
window.ABTestCraft.on('conversion', (event) => {
  console.log('Conversion tracked:', event.goalType);
});

Manual Initialization

If you're loading scripts asynchronously:

javascript
window.ABTestCraftReady = function(abtestcraft) {
  abtestcraft.trackConversion('page_loaded');
};

GA4/GTM Integration

ABTestCraft integrates with Google Analytics 4 and Google Tag Manager via the dataLayer.

Enable Integration

In Settings → ABTestCraft, enable GA4 Integration.

DataLayer Events

When enabled, the following events are pushed to the dataLayer:

#### Test Participation

javascript
{
  event: 'abtestcraft_participate',
  abtest_name: 'Homepage Hero Test',
  abtest_variant: 'variant-a',
  abtest_id: 'test-123'
}

#### Conversion

javascript
{
  event: 'abtestcraft_conversion',
  abtest_name: 'Homepage Hero Test',
  abtest_variant: 'variant-a',
  abtest_goal: 'form_submission',
  abtest_id: 'test-123'
}

GTM Configuration

In Google Tag Manager:

  1. Create a trigger for abtestcraft_participate events
  2. Create a trigger for abtestcraft_conversion events
  3. Send these as GA4 events with the relevant parameters

Console Commands

ABTestCraft provides console commands for maintenance and data management.

List Tests

View all tests with their status (draft, running, paused, completed):

bash
php craft abtestcraft/list

View Statistics

Display test statistics including impressions, conversions, and significance:

bash
php craft abtestcraft/stats
php craft abtestcraft/stats --test=123

Recalculate Statistics

Clear cached stats and recalculate from raw data:

bash
php craft abtestcraft/recalculate-stats
php craft abtestcraft/recalculate-stats --test=123

Cleanup Old Data

Remove visitor and conversion data older than the retention period:

bash
php craft abtestcraft/cleanup

Export Results

Export test results to CSV:

bash
php craft abtestcraft/export --test=123
php craft abtestcraft/export --all

Permissions

ABTestCraft integrates with Craft's user permission system.

Available Permissions

  • Manage tests (`abtestcraft:manageTests`): Create, edit, start, pause, and delete A/B tests
  • View results (`abtestcraft:viewResults`): View test results, statistics, and analytics

Assigning Permissions

  1. Navigate to Settings → Users → User Groups
  2. Select a user group
  3. Under ABTestCraft, check the desired permissions
  4. Save the user group

Example Configuration

For a "Marketing" group:

  • ✅ Manage tests — Can create and run tests
  • ✅ View results — Can see performance data

For a "Content Editor" group:

  • ❌ Manage tests — Cannot modify tests
  • ✅ View results — Can view test performance

Best Practices

Follow these guidelines for effective A/B testing.

Statistical Significance

  • Wait for significance: Don't end tests early. Wait until you reach your confidence threshold (typically 95%).
  • Adequate sample size: Run tests long enough to gather sufficient data. Small sample sizes lead to unreliable results.
  • One change at a time: Test single variables to understand what drives the difference.

Test Design

  • Clear hypothesis: Define what you're testing and why before starting.
  • Meaningful variations: Make changes significant enough to potentially impact behavior.
  • Document everything: Keep records of what you tested and learned.

Technical Considerations

  • Test the test: Preview both variants before going live.
  • Monitor for errors: Check for JavaScript errors or broken layouts in variants.
  • Consider caching: If using full-page caching, ensure variant assignment happens before caching.

Common Mistakes to Avoid

  • Ending tests too early based on "promising" results
  • Running too many tests simultaneously on the same page
  • Testing on pages with very low traffic
  • Making changes to variants mid-test

Data Retention

ABTestCraft stores test participation and conversion data. Configure retention to balance analytics needs with privacy requirements.

Default Retention

By default, data is retained for 90 days after a test ends.

Configuring Retention

Set the retention period in Settings → ABTestCraft:

  • 30 days: Minimum for most testing needs
  • 90 days: Recommended for thorough analysis
  • 365 days: Extended retention for long-term comparison

Manual Cleanup

Run the cleanup command to remove old data:

bash
php craft abtestcraft/cleanup

GDPR Compliance

ABTestCraft uses first-party cookies only and doesn't share data with third parties. For GDPR compliance:

  1. Include A/B testing in your privacy policy
  2. Consider cookie consent integration
  3. Use the data export feature for subject access requests

Troubleshooting

Common issues and how to resolve them.

Tests Not Starting

Symptoms: No visitors are being assigned to variants.

Solutions:

  1. Verify the test is set to "Active" status
  2. Check that the test start date has passed
  3. Ensure the control entry is published and accessible
  4. Clear any full-page caches

Conversions Not Tracking

Symptoms: Visitors participate but no conversions appear.

Solutions:

  1. Check the JavaScript console for errors
  2. Verify goal configuration matches your tracking code
  3. Ensure the ABTestCraft JavaScript is loaded
  4. Check that cookies are not being blocked

Variant Not Showing

Symptoms: All visitors see the control version.

Solutions:

  1. Check that the variant entry is published
  2. Verify variant assignment in browser cookies
  3. Test in an incognito/private window
  4. Check for template caching issues

Statistical Calculations

Symptoms: Significance never reached despite high traffic.

Solutions:

  1. Ensure conversion rates differ meaningfully between variants
  2. Check for technical issues causing data loss
  3. Consider if sample size is truly adequate
  4. Review goal tracking accuracy

Getting Help

If you continue to experience issues:

  1. Check the [GitHub Issues](https://github.com/livehanddotcom/abtestcraft/issues)
  2. Email [[email protected]](mailto:[email protected])