A/B Split Testing on iOS

Mobile is growing up and as a result more and more engineering convention is growing around it. One area that there has been very little focus on (which is surprising given the popularity of the lean movement and ‘experiments’) is running dynamic experiments / A/B tests on mobile.

There are challenges. Lots of them, including a general belief that Apple is against dynamically changing the UI on the iPhone, thus preventing tests. Nowhere in the developer guidelines does it say ‘Apps that change the colour of a button to test performance and report in this performance will be rejected’. The guidelines are there to protect against bad software, experiences and malicious actors, split testing is designed to make better software and better experiences.

What would a mobile A/B test look like?

This was where I started from and the vague requirements that I came up with are as follows:

• The iPhone App can contact a server to get a set of test cases with possible values
• Each control that is a ‘test case’ should check this set of tests to see if there is a value in there for it
  • If so, set the value (text, image URL, color etc..) to the value of the test. Fix the value for that client so that one user doesn’t see multiple test cases.
  • If not, fall back to some local default
• When a positive interaction is completed, this is uploaded to the server with ‘test case’, ‘value’, ‘outcome (+/-)’
• Same for a negative interaction (e.g. Cancel button, or App backgrounding)
• These results are available on the server and used to inform the next set of tests

What about things that aren’t controls?

There are plenty of things to test that aren’t controls, e.g.:

• Conditions for asking to rate – (rate button being positive and ‘never’ being negative)
• Message text on an upgrade reminder
• Determinate vs. indeterminate progress loaders for content (bounce on loading, rate)
• etc…

For these cases, we want the ability to generically create a ‘test case’ and use it’s value, then post back outcomes within the Application logic.

The iOS A/B Split Test Library

I made this library over the weekend, so it has many shortcomings, but it allows you to perform A/B tests on some extended controls and generic tests using values downloaded from the PHP Server companion project.

• A/B Test UIButton text values using ABTestButton (configure using User Defined Attributes in Interface Builder)
• A/B Test UIButton images using ABImageTestButton (again all in Interface builder)

These will both automatically post positive outcomes when the user taps on them.

• A/B Test any generic values using ABTestCase

ABTestCase

ABTestCase allows you to create a test case with a token (as configured in the PHP Server) and get a test case value then use it however you see fit, then post the outcomes whenever you like.

To get a test value, create a new test case:

ABTestCase *testCase = [[ABTestCase alloc] initWithTestCase:@"YOUR_TEST_CASE_TOKEN"
    andControlValue:@"Your Default Text Value"];
NSString *testCaseText = [testCase value];

Then, when you want to send an outcome, use ABTestCaseOutcome

ABTestCaseOutcome *outcome = [[ABTestCaseOutcome alloc] initWithTestCase:@"YOUR_TEST_CASE_TOKEN"
   andOutcomeResponse:ABPositiveResponse];
[outcome send];

The code is available for GitHub under the MIT License – I hope that it’s a useful start to your A/B testing efforts in mobile – I’d love to hear any comments. Also, any enhancements are more than welcome!!

iOS A/B Test Library on GitHub{.woo-sc-button.silver.large} A/B Test PHP Server{.woo-sc-button.silver.large}