by David M. Raab
DM News
December, sales 2006
.
Yes it’s a cliché, but testing truly is the heart of direct marketing. Yet your grandfather’s A/B split is just as obsolete as his Philco radio. Today’s direct marketers use a more sophisticated technique, known as multi-variate testing, to evaluate many factors simultaneously and identify the best possible combination—even if it’s one that was never actually tested.
Actually, Grandpa might feel right at home with multi-variate testing, since the concepts were originally developed in the 1920’s and adopted for industrial use in the 1950’s. Marketing applications date from the late 1990’s. In all cases, the general idea is the same: an experiment can measure the impact of several elements, each with multiple versions, by testing a small fraction of all possible combinations. Results for each element are read separately: so all results for headline A are compared with all results for headline B, even though some people saw different copy, prices, photos, and so on. Results from different combinations of elements are estimated by adding up the impacts of their components. (This is a simplified explanation: there are additional nuances that only a statistician could love.)
Offermatica (Offermatica, Inc., 866.627.3557, www.offermatica.com) offers both multi-variate and A/B testing for Web marketers. The system not only helps marketers design the tests, but executes them by taking over specified areas on a Web page and controlling their contents for each site visitor. The execution functions are critical because the mechanics of test delivery are much more demanding than the test design itself.
Each multi-variate test in Offermatica starts with a test name, start and end dates, and percentage of visitors to include. Users can optionally specify targeting rules to narrow the audience and definitions of segments within the audience to report on. The user then builds a list of elements and versions, which the system automatically converts into a test grid. Finally, the user links the elements to Web page locations, which Offermatica calls “mboxes”.
The mboxes are physically added to Web pages by inserting a couple lines of Javascript. These send the mbox name to an Offermatica server and display content that Offermatica returns. Each mbox has its own Javascript; all code is identical except for the mbox name. The mbox can also transmit the page ID, visitor ID, and URL parameters such as search terms. These can be used to update visitor profiles and capture test outcomes such as clicking on an ad or placing an order. Users can also track outcomes by importing external data such as order logs. Visitors are identified by first party cookies which contain an ID linked to a detailed profile stored at Offermatica.
Offermatica automatically builds its list of available mboxes by adding each one when it first calls the server—that is, the first time the page with the mbox is viewed in any browser. This usually happens immediately after the Javascript is added to a Web page.
Adding mboxes is pretty much the only involvement that Web site technicians have with Offermatica. Test content can be uploaded to Offermatica using the system interface, stored on a client server or reside with a third party. When the content is elsewhere, Offermatica stores an identifier that tells the other system what to deliver. Users can view the each piece of content within the Offermatica interface and can preview full Web pages as contents are assigned to mboxes.
Reports show both the winning version for each element and the winning combination that was actually tested. Results are updated in real time. Users specify the success metric, with options including conversion rate, lift, average order value, revenue per visit and total sales. The system also shows the influence and confidence level of each result. Users can filter results by segment, week day vs. week end, and time period, and can exclude very large orders that might skew the results. A ‘push winner’ button makes the winning combination the default for all visitors in one step. If the best combination was not tested, users must set it up manually.
The system does not identify interactions or correlations among test elements. Users looking for interactions can download element-by-element statistics or have Offermatica staff explore the data for them. Offermatica generally recommends that tests be designed so that interactions are not a major concern, arguing that quick, simple tests are ultimately more productive than larger, more complicated ones.
But in practice, Offermatica places very few constraints on its users. The system does not limit the number of variables or elements per test. The same mbox can be used by multiple tests and appear in multiple locations. Any content can be assigned to any test or mbox. Visitors can be kept within the same test over multiple visits or not. Users can set priorities across tests and apply target segments within tests to control how such conflicts are resolved. This flexibility makes the system very powerful, although it also opens opportunities for error. Offermatica account managers and consultants help users make the right decisions and interpret their results.
Because Offermatica controls the mbox content seen by all visitors, it can do more than testing. One approach is to set rules that deliver different content based on the visitor’s source, site behavior or profile. This could, for example, treat existing customers differently from prospects or make offers related to previous purchases. Another approach uses “self-optimizing” tests that automatically increase the proportion of visitors shown the best-performing combination as the test progresses. Such tests review results every two hours, so they can adjust to changes in user behavior over time. A new offering, called “AdBox”, manages online ads served outside the client’s own site.
Offermatica is sold as a hosted service. Contracts run for one year or more and range from $5,000 to $25,000 per month based on the volume of test visits and staff hours. The original version of the product was released in 2004 and the company states it has more than 100 active customers.
Leave a Reply
You must be logged in to post a comment.