2006 Dec 01
Offermatica, Inc. Offermatica
by David M. Raab
DM News
December, 2006
.
Yes it’s a cliché, but testing truly is the heart of direct marketing. Yet your grandfather’s A/B split is just as obsolete as his Philco radio. Today’s direct marketers use a more sophisticated technique, known as multi-variate testing, to evaluate many factors simultaneously and identify the best possible combination—even if it’s one that was never actually tested.

Actually, Grandpa might feel right at home with multi-variate testing, since the concepts were originally developed in the 1920’s and adopted for industrial use in the 1950’s. Marketing applications date from the late 1990’s. In all cases, the general idea is the same: an experiment can measure the impact of several elements, each with multiple versions, by testing a small fraction of all possible combinations. Results for each element are read separately: so all results for headline A are compared with all results for headline B, even though some people saw different copy, prices, photos, and so on. Results from different combinations of elements are estimated by adding up the impacts of their components. (This is a simplified explanation: there are additional nuances that only a statistician could love.)

Offermatica (Offermatica, Inc., 866.627.3557, www.offermatica.com) offers both multi-variate and A/B testing for Web marketers. The system not only helps marketers design the tests, but executes them by taking over specified areas on a Web page and controlling their contents for each site visitor. The execution functions are critical because the mechanics of test delivery are much more demanding than the test design itself.

Each multi-variate test in Offermatica starts with a test name, start and end dates, and percentage of visitors to include. Users can optionally specify targeting rules to narrow the audience and definitions of segments within the audience to report on. The user then builds a list of elements and versions, which the system automatically converts into a test grid. Finally, the user links the elements to Web page locations, which Offermatica calls “mboxes”.

The mboxes are physically added to Web pages by inserting a couple lines of Javascript. These send the mbox name to an Offermatica server and display content that Offermatica returns. Each mbox has its own Javascript; all code is identical except for the mbox name. The mbox can also transmit the page ID, visitor ID, and URL parameters such as search terms. These can be used to update visitor profiles and capture test outcomes such as clicking on an ad or placing an order. Users can also track outcomes by importing external data such as order logs. Visitors are identified by first party cookies which contain an ID linked to a detailed profile stored at Offermatica.

Offermatica automatically builds its list of available mboxes by adding each one when it first calls the server—that is, the first time the page with the mbox is viewed in any browser. This usually happens immediately after the Javascript is added to a Web page.

Adding mboxes is pretty much the only involvement that Web site technicians have with Offermatica. Test content can be uploaded to Offermatica using the system interface, stored on a client server or reside with a third party. When the content is elsewhere, Offermatica stores an identifier that tells the other system what to deliver. Users can view the each piece of content within the Offermatica interface and can preview full Web pages as contents are assigned to mboxes.

Reports show both the winning version for each element and the winning combination that was actually tested. Results are updated in real time. Users specify the success metric, with options including conversion rate, lift, average order value, revenue per visit and total sales. The system also shows the influence and confidence level of each result. Users can filter results by segment, week day vs. week end, and time period, and can exclude very large orders that might skew the results. A ‘push winner’ button makes the winning combination the default for all visitors in one step. If the best combination was not tested, users must set it up manually.

The system does not identify interactions or correlations among test elements. Users looking for interactions can download element-by-element statistics or have Offermatica staff explore the data for them. Offermatica generally recommends that tests be designed so that interactions are not a major concern, arguing that quick, simple tests are ultimately more productive than larger, more complicated ones.

But in practice, Offermatica places very few constraints on its users. The system does not limit the number of variables or elements per test. The same mbox can be used by multiple tests and appear in multiple locations. Any content can be assigned to any test or mbox. Visitors can be kept within the same test over multiple visits or not. Users can set priorities across tests and apply target segments within tests to control how such conflicts are resolved. This flexibility makes the system very powerful, although it also opens opportunities for error. Offermatica account managers and consultants help users make the right decisions and interpret their results.

Because Offermatica controls the mbox content seen by all visitors, it can do more than testing. One approach is to set rules that deliver different content based on the visitor’s source, site behavior or profile. This could, for example, treat existing customers differently from prospects or make offers related to previous purchases. Another approach uses “self-optimizing” tests that automatically increase the proportion of visitors shown the best-performing combination as the test progresses. Such tests review results every two hours, so they can adjust to changes in user behavior over time. A new offering, called “AdBox”, manages online ads served outside the client’s own site.

Offermatica is sold as a hosted service. Contracts run for one year or more and range from $5,000 to $25,000 per month based on the volume of test visits and staff hours. The original version of the product was released in 2004 and the company states it has more than 100 active customers.

* * *
David M. Raab is a Principal at Raab Associates Inc., a consultancy specializing in marketing technology and analytics. He can be reached at draab@raabassociates.com.

2006 Nov 01

Vtrenz Inc. iMarketing Automation
by David M. Raab
DM News
November, 2006

Sales and marketing organizations have been among the most enthusiastic adopters of hosted software (also known as on-demand, software as a service, or application service providers), which allows companies to use systems that are run for them by an external vendor. This is particularly true at smaller firms, where technical resources tend to be stretched thin and where sales and marketing systems can be effective without extensive customization or integration with the other company processes.

Many hosted sales and marketing systems resemble their on-premise counterparts: they provide sales force automation, customer service, marketing management, or several of these combined. But some don’t fit into those standard categories. Consider Eloqua Conversion Suite (www.eloqua.com), reviewed here in May 2006, and Vtrenz iMarketing Automation (www.vtrenz.com), reviewed below. Designed primarily for lead generation and nurturing, they combine campaign management, email, and Web execution with links to external sales and service software. They differ from conventional “enterprise marketing management” in using a simple database structure—typically a contact profile supplemented by interaction history—rather than a complete marketing database with extensive transaction details. They also lack advanced budgeting, project management, and analytics capabilities of a full-fledged enterprise marketing system.

Vtrenz iMarketing Automation (Vtrenz Inc., 701-478-7704, www.vtrenz.com) is particularly impressive for its campaign capability. Campaigns can include multiple tracks, each with a sequence of steps. Entry criteria determine which contacts move into each track and each step. Each step has its own execution schedule, which can be a fixed date or relative to the time a participant reached the preceding step. Each step also has an action, such as receiving a particular email message, that is applied to the people who reach it.

Users can also define criteria to move prospects from their current track to the start of another track or campaign. This allows the system to react to prospect behavior or profile changes after the campaign begins. Additional criteria at the campaign, track, and step levels can send prospects to an external destination such as a sales automation system. Users can define business logic to choose the recipient of the exported records. They can also set the timing, data elements and an accompanying message.

None of this is particularly glitzy: the tracks and steps are simply listed in sequence rather than displayed on a graphical flow chart. But it does let non-technical users define complicated treatment processes that adjust to each prospect’s behavior. Since the system can automatically migrate customers from one campaign to another, it could in theory execute a completely automated set of contact programs throughout the customer life cycle.

All types of selection criteria are created with the same user interface. Users define one or more rules, each having one or more statements. The statements are built by selecting from a list of categories including database tables and campaign resources such as email, direct mail, surveys and Web sites. Users then select from a list of statements appropriate to the category, such as having recently clicked on a link in an email. Finally, they apply parameters such as the specific link and time period to check. Statements can draw on profile data, campaign participation, or contact behavior. Statements are linked with ‘and’ or ‘or’ conditions.

This approach allows non-technical users to build complicated statements and presents the result in an English-like format. But it is limited to the 250+ statements that the vendor has predefined: even technical users cannot add their own.

Contacts for a one-shot campaign can be selected by creating a list, while on-going campaigns can execute their entry criteria on a user-defined schedule. On-going campaigns also have start and end dates and rules for whether the same person can receive the campaign more than once. Users can specify the hours of the day and days of the week during which the campaign is active, typically to limit responses to normal business hours.

Campaign actions are associated with resources including email, direct mail, fax messages, and Web pages, surveys and forms. Users can create these with the system’s HTML generator, which has an interface similar to Microsoft Word, or they can import HTML created elsewhere. Messages can be personalized and forms pre-populated by pulling information from the underlying database. Emails can also contain links to personalized Web pages. By the end of 2006, system-generated pages will be able to include business logic to display different message blocks based on contact profiles or behavior.

Survey and other form results can be posted to the contact database—which is limited to 40 standard fields and 40 user-assigned fields—or a separate survey response table. Users can publish or disable campaign resources and see which campaigns use them. A separate “assets manager” catalogs elements such as graphics files.

Contact names enter the Vtrenz database through file imports, system-generated Web forms, or automated data exchange with sales automation systems. Vtrenz has a Application Program Interface (API) to accept such data, plus a specialized API tailored to Salesforce.com. Another API is planned for Microsoft CRM. Duplicate identification is limited to exact matches on full or partial strings.

The Vtrenz database contains the 80-field contact table, survey table, and history of campaigns, messages, and responses. Vtrenz assumes more detailed information, such as account data and purchase history, will reside in other company systems and be imported in summary form as needed. “Extension tables” can store more data within Vtrenz, but take custom set-up by Vtrenz staff.

The system provides basic email delivery, bounce and click-through statistics. It tracks survey and microsite visits to measure response, but lacks integrated campaign analysis or A/B split testing. Vtrenz offers a Web tracking service based on first party cookies and code snippets embedded in a Web page.

Vtrenz was founded in 1999 and currently serves more than 250 clients. Most are business to business marketers. Pricing is based on the number of profile names, users and system features. The simplest version starts at $10,000 per year plus $5,000 for set-up and implementation, although most clients spend more to include additional options.

* * *

David M. Raab is a Principal at Raab Associates Inc., a consultancy specializing in marketing technology and analytics. He can be reached at draab@raabassociates.com.

2006 Oct 01

DecisionPower Inc. MarketSim
by David M. Raab
DM News
October, 2006

Agent-based modeling uses computer programs to simulate quasi-independent entities as they interact with their environment and each other. It is frequently used to model things like ecosystems and network traffic, although it is probably most familiar from computer games such as SimCity. In an agent-based model, the actions of each entity (agent) are determined by rules that process inputs including the agent’s own state (“hungry”), external factors (“food on that rock”), and other agents (“crowd around the food”). Results over time can produce complex patterns of behavior that are not explicitly defined in the rules themselves. In the example I just gave, the creature might balance how hungry it is, the distance to the food, and the size of the crowd to decide whether to move towards or away from the rock.

MarketSim (DecisionPower Inc., 408-379-9200, www.decisionpower.com) applies agent-based modeling to consumer behavior. In particular, it is used to predict results in a product category such as snack foods or headache remedies. In a model of this sort, the primary agents are consumers, while the environment includes brands, products and channels. Rules describe how consumers make purchase decisions, taking into account factors such as product attributes, consumer preferences, marketing activities, distribution, and external influences like the weather. When the model is run with specific inputs such as particular set of products and marketing activities, the system simulates what consumers will buy over time and generates outputs for market share, revenue, profits, unit sales, and other business measures.

This may sound quite simple, but in practice it is not. MarketSim’s must match the results of actual consumer markets with enough accuracy for businesses to determine the likely results of a particular marketing plan or product launch. This means models all the competitors in a product category, as well as the different distribution channels and types of customers. Each has its own characteristics.

The most complex entity is the consumer. MarketSim provides over 100 prebuilt rules, of which 25 or so might be used in any particular model. These rules describe how customers make their purchase decisions, taking into account their preferences for different product attributes; information gathered from advertising and personal contacts; consumption volume and frequency; shopping behavior such as channels used and responsiveness to in-store displays and price differences; responsiveness to coupons; and previous experiences. Each model includes multiple customer segments with their own settings for rules such as the weight assigned to different product attributes.

The number of agents assigned to a segment would be proportionate to the size of that the segment in the actual marketplace, although the total number of agents need not equal the number of actual customers. A market of many millions could be accurately modeled with 100,000 to 150,000 agents.

A model would also include dozens or even hundreds of specific products, depending on the complexity of the marketplace and the degree of detail required. Products are linked to brands. Each brand will have its own attributes, which use the same categories as consumer preferences. This provides the connection between brands and consumers that is needed to model consumer choice. Each brand also has a marketing plan with price, display, distribution, media, and coupon details for each time period.

All these rules and attributes must be set so they result in an accurate prediction. DecisionPower does this by gathering historical data—typically three year’s worth—for actual sales, marketing activities, product attributes, distribution, and external factors. This data covers all competitors, not just the model sponsor. It is fed into the model and the rules are tuned until the system gives acceptably realistic results. Modelers usually feed in two and a half years of actual data and then compare simulated results for the final six months with the known actual results for the same period.

This calibration process occupies most of the three to four months it takes to build a major MarketSim model. Most of the work involves adding new data sources or events to reduce anomalies in results. A calibrated model can give reliable predictions about one year into the future, and will remain valid—assuming the inputs such advertising spend are updated—for one or two years before it must be rebuilt.

Constructing a MarketSim model is largely a task for experts, either employed by DecisionPower or in the research department of a client. The system offers a graphic user interface that is more than adequate for such purposes. A separate module called BrandManager lets non-technical users specify model inputs, such as alternative marketing plans, on an Excel spreadsheet. BrandManager then imports the spreadsheet, loads the data into MarketSim proper, and runs a model with the new assumptions. Users can save their inputs as scenarios, allowing them to easily compare different options. They can view results on the screen, selecting the scenarios, products and measures to compare, and displaying them as a table or graph. Results can also be exported back to Excel for further analysis.

MarketSim was introduced in 1996 and has been the primary focus of DecisionPower’s business since 2001. The end-user portion of the system runs on a Windows PC, while data—which can be quite voluminous because of the all the historical detail—is typically stored on a central server. Small models can run on the end-user system but larger ones are often sent to a separate modeling server. DecisionPower offers hosted options for clients that prefer not to install the system in-house. Running a single scenario may takes about ten minutes for a simple model and 45 minutes to an hour for a very complicated one. Companies often run multiple scenarios—sometimes hundreds—as they explore alternatives to identify optimal marketing strategies.

DecisionPower charges from $100,000 to $300,000 to develop a MarketSim model, depending on the project. Clients can license the completed model to run in-house for $100,000 per year. DecisionPower has sold 60 to 80 MarketSim models to date, mostly to consumer package goods manufacturers. A lower cost option, MarketSimExpress, is available for quicker, simpler projects.

* * *

David M. Raab is a Principal at Raab Associates Inc., a consultancy specializing in marketing technology and analytics. He can be reached at draab@raabassociates.com.