Privacy concerns have grown from a distant cloud on the bright horizon of relationship marketing to a looming thunderhead that may burst any second. The immediate catalyst has been the Internet, where consumers are acutely aware that their every movement can be tracked, recorded and analyzed. But database marketers have collected reams of personal data for years, a practice that generated fewer complaints only because consumers were less familiar with the details. Now that the Internet has made privacy a public issue, all kinds of data gathering are being examined.

Many marketers are genuinely perplexed by consumers’ concern. While marketers are often willing to play on public distrust of government snooping when it serves their purposes, they see little potential for abuse of data held in private hands. After all, a private business can’t put you in jail or make you pay taxes. The commercial reason for gathering personal information is benign: to provide advertisements and services that are tailored to individual interests. So long as consumers have the option to reject any offers they receive, what’s the harm? As these marketers see it, the whole issue has been manufactured by a handful of privacy nuts who are either genuinely paranoid or cynically use privacy to further their own political agendas. Proponents of this view point to opinion surveys, low use of existing opt-out services, and consumers’ willingness to trade data for slight compensation as evidence that the real value people place on data privacy is quite low.

But such arguments miss the point. True, most people don’t care about trivial annoyances, such as advertising, and therefore won’t pay even a low price in effort or cash to avoid them. But personal data gathered by businesses can be used for decisions with considerably greater impact than ad messages. Remember, the fundamental goal of relationship management is to tailor all interactions to build the optimal relationship with each customer. This goes beyond advertising to include which products are offered, how they are priced, and what level of service is provided.

This sort of data-driven tailoring is usually described as treating customers differently, but a less polite way to put it is that some will be treated better than others. Think about the airlines: your status with the company determines everything from how quickly they answer the phone to take your reservation, to how long you wait to check in, to how soon you can board, to when your luggage comes off, in addition to mention your legroom, food and drinks during the flight itself. Interestingly, this particular set of privileges seems to attract little hostility from the have-nots, perhaps because they still perceive air travel as a luxury. Compare this to the anger that banks generate when they charge additional fees or limit the services available to low-balance customers. This type of discrimination is perceived as hugely unfair, presumably because banking is considered an economic necessity, or perhaps because it seems to target lower income customers.

Although the superior treatment that airlines, banks and others give their best customers raises some issues of fairness, it is not primarily a privacy issue because the distinctions are based on transactions between the customer and the business itself. That is, there is no hidden data collection and no sharing of data from other sources. There are also no inferences based on predictive models or scorecards: the criteria are objective measures such as miles flown or balances maintained. This means that even though a policy may be inherently unfair, it is at least accurately and consistently applied.

By contrast, the worst privacy nightmares involve data that may be collected surreptitiously, inappropriately shared with others, applied incorrectly or just plain wrong. This is where the real harm gets done: someone is denied a job because they visited a gay Web site; someone is charged a higher health insurance premium because the insurer sees their past medical bills; a loan application is rejected based on a defective statistical model; a service request is denied because an external database suggests the buyer will not be a good future customer.

A privacy skeptic might point out that most of these situations involve a problem unrelated to privacy itself: people shouldn’t be penalized for visiting unpopular Web sites, people shouldn’t lie about their past medical bills, statistical models shouldn’t be defective, data shouldn’t be incomplete or inaccurate. But such problems do exist and always will; it would be silly to ignore them, assume they will all be fixed, or pretend we can legislate them out of existence. They are part of the privacy issue because they only hurt people when large amounts of personal data are widely available: otherwise, no one would be able to check what Web sites someone had visited, look at their medical bills, use certain variables in statistical models, and so on. In effect, privacy wraps a blanket of ignorance around each individual that prevents companies from even trying to discriminate among them, for good reasons or bad. It seems likely that a visceral understanding of the protection provided by privacy underlies most people’s concern for its loss.

Of course, no one has absolute privacy, and giving up information confers benefits as well as costs. So the privacy debate is really about striking a balance between the value that data can provide and the problems that it can cause. It’s tempting to argue for a free market solution, of letting individuals negotiate with companies about which data to share, for what uses and with what compensation. But things aren’t so simple: if most people decide to share a particular piece of information, then those who do not may be falsely assumed to be hiding something, or simply lumped into a category that gets worse treatment than others. So even voluntary data sharing results in effective coercion of those who would prefer to opt out. This means, paradoxically, that only government regulations can make data sharing truly voluntary, by protecting those who choose not to share.

Obviously there are economic and social costs to such regulation, so it should be limited to types of data that are worth controlling. The government also needs to ensure that companies use data only in the agreed ways, just as it would enforce any other contract. Finally, society may decide that some types of data should never be collected or should only be used for particular purposes. So, free market fantasies notwithstanding, there is no alternative to government involvement in this area. In reality, extensive government regulation already exists in areas such as credit reports and medical information.

The real question, then, is not whether the government should be involved in privacy regulation, but how. This ultimately depends on the social aims the regulation supports. One aim is privacy itself–the sense that what a person does should be nobody else’s business unless there is a good reason otherwise. Today such a right is widely, and legally, acknowledged, although it is still questioned in some circles. Another fundamental social goal is fairness: the idea that everyone should be treated equally unless they are different in a significant, relevant way. In today’s United States, differences such as race and religion are almost never acceptable grounds for differential treatment; other differences, such as income and education, are suspect but not automatically forbidden. Fairness is justified as a fundamental moral imperative–it is simply the right thing to do. In addition, fairness has a practical justification: giving everyone equal opportunities helps to ensure that society gets the benefit of all of its members’ talents.

Since the bedrock principle of fairness is that people should be treated equally unless there is a reason not to, any differential treatments–including the treatments of relationship management–need some justification before they are accepted. In the airline and banking examples mentioned earlier, the justification is fairly easy: the differences are based on relevant past behavior by the specific individuals involved. But what about differential treatment that is not based on concrete information about specific customer transactions? If the information is likely to be inaccurate or lead to false conclusions, it is considerably less defensible because people may be discriminated against unfairly. And–here’s where privacy comes back in–much of the personal data that is surreptitiously shared among companies is prone to exactly this sort of problem. There may be errors in the data itself, errors in matching data to the right person, approximations used in place of real data, and any number of other flaws. Nor, unfortunately, are the errors likely to be unbiased: data based on averages will penalize people who belong to disadvantaged groups and reward those who belong to more privileged strata, regardless of their personal characteristics. This means that differential treatment based on inadequate data is not only immoral, but diminishes the social mobility that is a key practical benefit of fairness. The utilitarian argument could be extended to oppose differential treatments even when they are based on accurate data: the theory is that treating everyone the same effectively subsidizes less well-off people, making it easier for them to ultimately succeed. But such subsidies are harder to defend on purely ethical grounds.

This is pretty abstract stuff, and there may be some holes in the logic. But what it boils down to is this: data privacy raises serious personal and social issues. It is not merely the irrational concern of a handful of paranoid Luddites. Relationship marketers should not fight blindly for the widest possible freedom in how they use personal data but should carefully seek to balance valid business and social considerations. Those who do may well find themselves supporting considerably greater restrictions on data sharing than they originally expected.

By all rights, service bureaus should be extinct. Burdened with high-cost, slow-moving mainframes, these dinosaurs of the marketing services world should long ago have been supplanted by more nimble, less costly in-house systems. The growing desire for real-time, company-wide integration between marketing and operational systems, which seems to require that the marketing database reside in-house, should have been the final nail in the coffin.

Yet service bureaus continue to prosper. The reason is simple enough: most firms lack the skills to build and maintain a serious marketing database themselves. Such skills have always been rare, and today’s shortage of all types of computer staff has made them still harder to find. Combine this with the risk of in-house development and the need to move quickly in an ever-more-competitive marketplace, and the bureaus’ promise to deliver a sophisticated system in a reasonable time at a controllable cost is nearly irresistible.

But even in the relatively stable world of marketing service bureaus, change does occur. One of the more interesting recent developments is a shift away from traditional volume-based pricing–where customers were charged on a per-thousand basis for every processing step–to a flat-fee model where customers buy a certain amount of hardware, software and staff capacity and are free to use it pretty much as they please. Although few firms have moved to a pure flat-fee model, many have moved in that direction. (In fact, a recent study by Raab Associates found only three of ten proposals relied exclusively traditional unit-based pricing; in half the proposals, over 50% of the fees were not volume-related.)

Part of the reason for the change is technical. Traditional mainframe technologies involved large computers whose capacity greatly exceeded the needs of most individual service bureau clients. The databases were typically maintained through large, periodic batch updates during which one client’s work took over most of the computer’s resources for a brief period of time. In this environment, it made sense to share one large computer among many clients, and to charge those clients based on the proportion of that computer’s capacity that they consumed. Pricing was set by finding some measure of utilization such as processor cycles, figuring how many units of that measure could be processed when the machine ran at its practical capacity, calculating the cost to operate the computer (including downtime and overhead), and arriving at a cost per unit to charge clients. The effect was to translate a largely fixed cost–maintaining a large mainframe computer–into variable unit costs. This approach may sometimes result in prices that do not reflect true underlying costs: if the computer is used at less than the expected load, costs are not fully covered; if demand so exceeds capacity that a new computer must be added, the incremental cost is much higher than the price charged the customer. But avoiding these dangers forces managers to pay close attention to capacity management, which is one of the keys to success in a high-fixed-cost environment. So, somewhat paradoxically, variable-cost pricing makes sense when you are running a fixed-cost mainframe.

But today’s service bureaus have increasingly moved away from mainframes to Unix or Windows NT-based servers. These systems cost much less per unit of capacity than mainframes, except perhaps at the highest end of the capacity scale. More important, the smallest individual systems are much cheaper than the smallest mainframes. This means it now does make sense to think of dedicating a single machine to an individual client. At the same time, clients are increasingly moving away from large periodic batch updates to smaller, more frequent updates–for example, daily instead of monthly. This removes the periodic spike in capacity demand that was the other major reason to use shared rather than dedicated machines to handle each client’s work.

But there’s more to this story than technology. At the same time that traditional marketing service bureaus are moving away from volume-based pricing, the most exciting new variation of the service bureau model is the application service provider or ASP. And guess what? Most ASP charges are based on volume.

The difference isn’t due to technology: nearly all ASPs run Unix or NT-based servers. And it isn’t due to usage patterns either: most ASP systems provide frequent if not real-time data access; few do large infrequent batch updates. Nor is there any fundamental difference in customer goals: people hire ASPs for the same reasons they hire traditional service bureaus, to get sophisticated systems running faster and more reliably than they could it themselves.

It appears the reason is a bit more subtle. Many ASP implementations are for operational systems, such as accounting or human resources management, or for production-oriented marketing applications such as outbound email campaigns or

Web site data analysis. This is a fairly sharp contrast to the marketing databases supported by traditional service bureaus: the exact use of these systems is often not understood when they are created and is expected to change over time.

In other words, the unstructured nature of a traditional marketing database makes it particularly suited to fixed cost pricing. In the quasi-operational world of ASP systems, each transaction has a fairly clear value: the marketer presumably can judge whether each piece of email is worth the incremental cost of sending it; the accounting people are comfortable with paying a little extra for each additional journal entry. But the value of a particular marketing analysis is just about impossible to predict, so there is no way for a marketer to justify the incremental cost of conducting it. This makes unit-based pricing particularly uncomfortable. Even worse, if the marketer does discover some valuable new application that involves much more intensive use of the database, volume-based pricing penalizes this success by driving up costs sharply. This is especially irksome because the marketer knows full well that the unit prices charged by the vendor are much higher than the true incremental costs of the added processing volume, so much of the cost increase is simply higher profit for the vendor.

The fixed price approach lets the marketer make judgements about how to allocate limited resources without facing the risk of sudden and unexpected changes in cost. This is a much more congenial environment for the experimentation and evolution that are the object of most conventional marketing databases. And, of course, if the marketer does find an application that significantly increases capacity requirements, there is still the ability to add hardware and support services in relatively small increments. So flexibility is retained.

Now that the distinction between structured and unstructured processing has been made, older….er, more experienced observers will also recognize that the structured processing done by ASPs resembles the tasks that service bureaus provided in the days before marketing databases: things like merge/purge and postal standardization. Of course, the service bureaus charged for these on a per unit basis, and they usually still do.

In short, while the switch from mainframe to server-based technology has something to do with service bureaus’ change from volume-based to fixed pricing, it is not only reason. Marketers who are evaluating vendor pricing schemes, or vendors who are designing such schemes, should also consider the nature of the task at hand. Volume-based pricing makes the most sense when the task is highly structured and well defined. Fixed pricing–that is, buying a bucket of capacity to be applied as the user pleases–makes more sense when the tasks and their values are less well understood.

One of the major challenges in building a customer-centric database has always been matching records from different systems that refer to the same customer. The necessary technologies–name and address standardization, postal costing, matching and householding–are fairly mature and quite familiar to firms with a long history of database marketing. But they are new to firms without this background and require a level of implementation skill that most will not have available in-house. Nor it is easy to add such skills, because experienced technicians are rare and because even if you hired one, they wouldn’t get enough practice at most companies to maintain their expertise. As a result, many firms either hire consultants to set up their systems or farm out the whole process to a service bureau.

Both of these approaches have drawbacks. Consultants can be expensive–though they are a bargain compared with the cost of failure–and may not train in-house staff to run the system after they leave. Service bureaus are even more expensive, and typically require days or even weeks to complete an update cycle. Perhaps worst in today’s time-pressed environment, it can take months to set up, test and refine a satisfactory in-house or bureau-based matching process.

Over the past few years, a new alternative has emerged. This involves using an external matching service on a real-time basis: that is, sending each new record to be processed as it is entered, and then loading the result directly into an in-house system. Vendors including Acxiom, Experian, Sagent, iMarket and Dun & Bradstreet all offer some version of this approach. While not necessarily cheaper than conventional methods, it offers near-immediate deployment, better quality than most firms could achieve themselves, and the option to simultaneously enhance the new record with external descriptive data.

External matching uses basically the same technology as conventional matching–that is, records are parsed into name and address components, standardized against postal and other reference tables, and then matched against other standardized records. Parsing and standardizing a single record takes just a fraction of a second, and poses no particular technical challenge. But matching has traditionally been a batch process that involved sorting the entire set of records to bring similar records together and then comparing neighboring records to each other. This won’t work with a real-time process, because each new record would have to be inserted into the sorted file as it was added, so future records could match against it. Moreover, maintaining a separate file for each client would be expensive and involve a time-consuming setup process.

Instead, external matching systems use a standard reference table of all individuals and households or businesses in the U.S. (or whatever market is being served). Such files, compiled from a variety of sources, are readily available to large service bureaus. Once an incoming record is standardized, it can be matched directly against this file regardless of what other records are in a particular company’s existing database and without any need to insert new records (except the handful that don’t match the reference table itself–which may or may not be worth adding). Each record in the reference file is assigned a unique, permanent ID. This means that when a match is found, the system merely appends this ID to the new record and returns it to the company, where it is easy enough to check whether a record with that same ID is already present in the company’s database. Thus, external real-time matching can be highly efficient from both a processing and storage point of view.

The reference table also lets the vendor pre-associate demographics and other enhancement data with each record, making it easy to return that data with the matched record itself. Similarly, the vendor can predetermine which individuals belong to the same household, so no on-the-fly processing is needed to return a household ID. The vendor can also use the permanent ID to link old and new addresses for the same individual, so if an old address is presented it can return the new address as well. Because the same reference table is used by many different clients, the vendor can afford to update it frequently, thereby providing even low-volume marketers with the most current information.

Unfortunately, assigning a standard ID to each indvidual also raises significant privacy issues. Since all match requests are processed against the same reference table, it would be easy enough for the vendor to keep track of which names have been presented by which sources–creating an unprecedentedly complete activity profile. Achieving the same result with conventional technology requires a massive matching job of many individual lists against each other, something that only very large marketers, compilers or name rental cooperatives can afford. The same technology would also let marketers subscribe to information about specified individuals–that is, be notified when a particular individual moves, makes a significant purchase, is reported to have an income change, or simply shows up on somebody else’s list. Again, conventional technology could only do this through periodic batch processes rather than continously.

The vendors of these services have policies that are designed to prevent what they consider privacy abuses, although some consumers might have different standards. The vendors also have strong economic motives to ensure that list owners do not use the standard IDs to share data without the vendor’s participation, and that list owners can use their services without fear their information will be used by others. So there are some safeguards built into the system. Whether these are ultimately deemed adequate remains to be seen.

In short, external matching provides an attractive alternative to conventional techniques in many situations. Marketers should be sensitive to privacy issues but take advantage when appropriate of its significant benefits.

“Dear Current: Picture yourself and the entire Resident family on a fabulous, all-expense-paid vacation….”

Such are the embarrassingly plebian ancestors of personalized marketing. From its early days as an attention-grabbing gimmick on pitches for magazines and shady real estate deals, personalization has grown in respectability to become the bedrock of modern customer management. Indeed, some would argue that personalization pretty much IS customer management.

Like any grand success, personalization has attracted hangers-on who are eager to use its popularity for their own purposes. Today, nearly every marketing-related system lists the ability to help plan, execute or analyze personalized interactions as one of its main benefits. (Come to think of it, those pitches themselves are rarely delivered in a personalized fashion. But let’s not dwell on what this might imply.)

Of course, marketers rarely bother to define their terms with any precision. As a result, so many conflicting claims being made about personalization that it faces an identity crisis. So, without getting bogged down in the theory of why personalization is important in the first place, it’s worth taking a look at what different kinds of personalization systems actually do. (For a range of opinions on some of the broader issues, try the excellent Web site at www.personalization.com.)

The most ambitious usage of personalization treats it as synonymous with interaction management–a process whose goal is to generate the most appropriate response to any customer action. Obviously the key to this definition is the meaning of “most appropriate”, which really boils down to the question of “most appropriate in terms of what?” That is, what factors are taken into consideration when trying to identify the most appropriate response?

Part of the answer has to do with measurement–what, exactly, are we trying to maximize? Is it short term profit? Net present value? Customer lifetime value? Retention rate? Market share? Customer satisfaction? Loyalty?

Personally, I favor a measure of return on investment–making sure that each dollar yields the greatest possible long-term profit. This isn’t particularly controversial, although it does mean accepting less-than-maximum profits from relationships with some customers. What’s really important is the definition treats every decision as an investment–even decisions that involve no incremental out-of-pocket expenditure, such as allocating limited resources within a call center or field force. How to actually perform this sort of measurement is a knotty problem. Fortunately, personalization can be deployed without solving it, so long as marketers are willing accept whatever imperfect approximations are available.

The other answer to “in terms of what” has to do with which factors are considered in selecting a reply. This is where today’s personalization systems vary most broadly.

The first factor to consider is information about the customer herself. This might seem pretty darn obvious, but in fact it’s possible to create “personalized” promotions that treat all customers the same–as in the “Dear Current” letter itself. It’s also possible to treat customers differently without knowing anything about them as individuals–consider a Web site that shows different pages depending on the visitor’s domain name or incoming link. Even systems that do identify individuals and treat them differently can vary significantly in the types of data they access. These range from a simple segment code appended during a batch process, to real-time look up of detailed historical information such as past purchase transactions, to access to information provided during the current interaction.

In fact, the breadth of available customer data is a significant differentiator among today’s customer management systems. For integrated Customer Relationship Management products like Siebel, a single customer transaction database is touted as a key advantage over single-purpose call center or sales automation systems that do not share data across touchpoints. Among Web-based personalization products, vendors who can integrate data from non-Web “legacy” systems take great care to distinguish themselves from products whose personalization is limited to profile data gathered within the system itself.

The second factor that systems might consider is the current interaction itself. Again, this may seem trivial: surely if the purpose of a personalization system is to respond appropriately to the current interaction, then the system must take that interaction into account? Yet, again, there are current products that do no more than distribute messages chosen in advance–say, by running a nightly batch process that produces a list of messages to be sent to individual customers if they should appear the following day.

This is nowhere as silly as it might sound. In fact, if each customer interaction is considered an opportunity to deliver a free advertisement, then delivering no message during those interactions would be as wasteful as buying a minute of TV time and broadcasting dead air. Viewed from this perspective, even a slightly off-target message is better than nothing–so long as some mechanism ensures the message isn’t so inappropriate in the context of the transaction that it actively antagonizes the customer.

Still, it does seem much smarter to deliver messages that are directly related to the current transaction. Obvious examples are recommendations for related products (cross sell) or more profitable alternatives (up sell). Many systems provide this sort of personalization, although they often develop the recommendations without considering the individual involved. For example, systems frequently present a list of products that are most often purchased together, without checking whether different recommendations might be appropriate for different customer segments, which products this particular customer has purchased or whether the customer has a history of accepting such suggestions at all.

If there’s no hope of a customer accepting the recommendation, then making it is a poor investment of company resources. Of course, the expected return on the investment depends on the expected revenue compared with the cost, and cost itself varies by channel: an email is much cheaper than the time of a call center operator. Then again, even a “free” Web message may carry a cost in terms of customer annoyance. As already noted, resolving this with absolute precision requires advanced metrics that do not really exist. But it’s still possible to set intelligent policies without those metrics–so long as the system provides access to the relevant data.

Systems also differ in exactly how much information they take into account about the current interaction, and in how broad a range of responses they consider. For example, some Web-based systems can distinguish the behavior patterns of a leisurely browser from that of a hurried buyer, and treat each appropriately. Or, to be a more precise, some Web-based systems capture enough data to make these distinctions based on rules defined by a marketer. Even the best of today’s Web-based systems couldn’t discover and react appropriately to these categories by themselves. Similarly, some personalization systems can consider a wide range of responses, such as whether it’s better to make a cross sell offer or to send the customer a thank-you note. This calls for more sophistication than just choosing the best cross sell offer, since it’s harder to find a common basis to use in ranking the alternatives. Once again, this is an issue of metrics and once again, today’s systems would mostly rely on rules created by marketers rather than on rankings the systems develop autonomously.

A third factor considered by some personalization systems is the customer’s status in current marketing campaigns. Many of today’s marketing systems assign customers to one or more campaigns that involve a sequence of multiple messages. Part of the challenge faced by any marketing system is to limit and coordinate these assignments so the customer is not overwhelmed by too many messages or sent messages that conflict. Like personalization, this poses the challenge of ensuring that the most productive of all possible choices are made. But campaign assignment is generally done from a campaign rather than customer orientation. (That is, the question is which customers to assign to a given campaign, not which campaigns to assign to a given customer.) This is not itself a personalization issue.

Where personalization and campaigns do intersect is when choosing the specific message to deliver during an interaction. If a customer is active in multiple campaigns, then the customer may be eligible to receive several different messages at the moment an interaction occurs. Since the system can presumably send just one message at a time (or, at least, needs to rank the different messages by priority), the personalization system must choose among them. This may be the knottiest of all the analytical problems associated with personalization, since it requires estimating not just the value of message by itself, but the incremental value of one message within a sequence. In fact, a truly thorough system would go further to estimate the value of repeating the same message compared with switching to a different message later in during the same interaction, and also somehow take into account the number and timing of future message delivery opportunities. It’s not even clear that a sound theoretical basis for making these decisions exists today. Certainly no personalization system can make such decisions in practice.

Still, decisions must be made. One common approach is to prioritize the campaigns when they are set up, using a single sequence that is the same for all customers. This has the virtue of simplicity, although it’s possible that a particular campaign will be of greater value for a specific customer than some other campaign that is ranked higher over all. In fact, it seems almost certain this will happen in some cases. When differences among the values of messages within the same campaign are added into the mix–for example, the sixth renewal offer to a magazine subscriber is worth less than the first–choosing solely on the basis of campaign priority seems inevitably suboptimal.

Some of these problems can be overcome through setting different priority sequences for different customer segments. If these segments are defined with enough precision, then the campaigns should be ranked correctly for almost every individual. Most systems that accommodate campaign rankings can be made to support this approach, although it might take some awkward tricks–such as creating several versions of the same campaign with different rankings and selection rules. A refined scheme along these lines is likely to be an administrative nightmare.

A more manageable approach would be to define campaign priorities on an individual basis. This could most easily be done when the customer is first assigned to a campaign. After all, if the system is well designed then it should already calculate the value as part of the selection process. Ideally the value would be recalculated over time. This could be done at fixed points during the life of the campaign–say, after every message–which would take into account the change in expected returns. Or, the value could be recalculated on-the-fly when messages need to be chosen. Systems that can execute scoring models in real-time can perform these sorts of selections. Sometimes the scores are used within a larger rule-based framework that makes broad choices–say, between retention campaigns and cross sell campaigns–before letting the system make the final specific selection. Since a customer may be eligible for many different campaigns, some sort of rule-based filter may in fact be necessary to limit the number of score calculations during each step in an interaction.

Still simpler methods of choosing among campaigns are also employed, and may even be appropriate when meaningful campaign values cannot be calculated. For example, systems that display Web page banner ads often look at a pool of messages for which a customer is eligible and select from this pool at random or in a rotating sequence. These systems often support general business rules such as the number of times a message can be displayed to a customer, the number of days each message is active, and the interval to wait before showing the same message again. Other approaches select campaign messages based on their scheduled delivery date–choosing whatever message is due soonest. This assumes the campaign system generates a list of future messages, which only some of them do.

Personalized messages also often vary depending on the medium in which the message is to appear. Ideally, this would be based on a measure of the effectiveness of different messages in different media. For example, a complicated offer that can be explained in detail on a Web page may be unsuitable for the one-line display of an automated teller machine. Similarly, some media can easily present multiple offers simultaneously while others cannot.

Few, if any, systems explicitly consider medium in their message selection mechanism. But separate versions of the same message must generally be prepared for different media, so marketers can implement a crude form of medium-based differentiation by simply not creating message versions for some media. Much of the value of this strategy depends on what happens when a message version is not available for the current medium. If the system goes back and selects the next-most-valuable message, there is a reasonable chance that something approaching maximum value will be achieved. But if the system simply presents a standard default message–or no message at all–the value is considerably less. Of course, the worst possible situation is if a missing message causes the system to crash altogether. Given the inherent complexity of managing an advanced personalization process, it’s essential that missing versions be handled smoothly and automatically. Of course, all this is a poor substitute for explicit measurement of message effectiveness–in a given situation, even a marginally effective version of the right message may be more valuable than the alternative.

This all assumes a central system is choosing messages to be delivered in different media. In fact, many systems today are limited to a single touchpoint such as Web pages or call centers. These systems often are highly sophisticated in some of their personalization abilities, but of course cannot coordinate messages across media. At best, it’s up to the marketers in charge to create rules and messages for different media that will generate similar outputs in similar situations. But the administrative effort required for such an approach makes it doomed under all but the simplest circumstances. This is a particularly important issue for systems that service just one side of the Internet/non-Internet divide. Unless customers conveniently limit themselves to one side or the other–which most won’t, given the option to use both–maintaining consistent customer management policies will likely prove impossible.

Channel workload is still another factor that some personalization systems consider. The most common examples are inbound telephone call centers, where managers routinely turn off options such as cross sell recommendations during busy periods to reduce the time spent on each call. Ideally, the system itself would decide the degree of personalization used for each call, balancing the value of maintaining a high service level with the value expected from extending a particular interaction. It’s quite likely that an accurate calculation would show that the value of personalization in some calls–say, where there is a very high probability of a lucrative incremental sale–outweighs the loss in revenues from other customers. As usual, a precise on-the-fly calculation is hard to develop but rules can provide a reasonable approximation. This is why elite members of frequent flyer programs have a special phone number for reservations.

Personalization can also be balanced against service levels at retail point of sale, kiosks, automated teller machines, and even high-traffic Web sites. Outbound marketing campaigns sometimes take into account other channel capacity issues, such as the number of leads that can be handled by a field force. In this case, the system has to determine which leads will yield the greatest value and automatically assign the rest to alternate channels such as telemarketing or email. This can be a tricky problem, since there may be other constraints such as target sales levels for individual products or a required mix of business for individual salespeople or dealers. In non-real-time situations, the system must also consider future capacity–it may make sense to hold off sending some leads to telemarketing today, if there is a high probability that a field sales person will have time to handle them tomorrow.

Finally, personalization systems must also adjust to other business considerations such as inventory positions, demand levels, and special promotions. For example, it makes little sense to cross sell a product in short supply–profits will be higher if the company offers other products and saves the product in limited supply for customers who request it by name. Conversely, it makes considerable sense to push a product that is overstocked, since it otherwise may end up as excess inventory. Airline yield management programs are one example of promotions that are dynamically adjusted to market conditions, although these are generally not tailored to individual customers. Better examples are programs that offer discounted tickets on excess seats to customers who have indicated an interest in flying to a particular city. While the details vary depending on the exact nature of the business consideration being accommodated, the general principle is that the personalization system needs to access information about that consideration and somehow factor this information into its value calculation.

By now, a few points should be clear. The first is that even though rule-based approaches are by far the most common today, they cannot manage the complexity of a system that considers all the listed factors–customer, current interaction, campaigns, medium, channel workload, and business constraints–in its personalization decisions. The only practical solution is a value calculation that produces a simple ranking of alternatives. While current value calculations are admittedly imperfect, they at least have the potential to be improved. By contrast, the rule-based approach is ultimately a dead end.

The second point is that there is no fundamental distinction between the personalization requirements for real-time, interactive systems and for outbound, batch-oriented campaigns. Both must make decisions based on a view of the entire customer relationship; in fact, each must take into account actions by the other. Similarly, there is no fundamental difference between “proactive” systems that scan transactions for opportunities and “reactive” systems that guide interactions initiated by the customer. Even though different technologies are needed to execute personalization through the different types of systems, they must share the same core decision functions.

A third point is that personalization systems must include mechanisms to help marketers refine their business strategies. These will range from standard test capabilities such as random sampling and champion/challenger comparisons, to automated data analysis, relationship discovery and model creation. They also include extensive storage of historical data, to allow the reconstruction of the conditions under which each interaction took place. Many of today’s personalization systems are very weak in these areas. But personalization cannot meet its promise without a systematic process to improve decision making. And as a practical matter, it will be virtually impossible to deploy such a process unless it is built into the personalization system itself.

The final point is that personalization systems will evolve into specialized products that are distinct from the operational systems which deliver the personalized messages. This is a substantial change from today’s situation, where most touchpoint systems provide their own personalization capabilities. Core capabilities of the new breed of personalization systems will include access to data from multiple sources, sophisticated value calculations, business strategy optimization, and communication with message delivery systems. Primitive versions of such solutions are beginning to appear, but it will be some time before a complete implementation is available.

Is campaign management a commodity?

Not so long ago, campaign management vendors competed on core capabilities: which system handled the most complicated promotions, included the most data, ran fastest, or was easiest to use. But over the past year or two, the terms of competition have shifted. Now vendors tout the scope of their offerings, citing integrated analytical reporting, email production, and statistical modeling. Basic campaign management itself–the ability to select names, keep a promotion history, and match responses to promotions–is almost an afterthought.

In some ways, this is a natural result of past industry trends. Over the past decade, systems that use standard or “open” relational databases like Oracle or SQL Server have steadily displaced products that use “proprietary” databases created by the campaign management vendors themselves. The main argument in favor of the open systems was that the data could be read by any number of third-party reporting and analysis tools. This eliminated a dangerous dependence on the campaign management vendor. But precisely because the “open” systems all use the same database engines and query language (SQL), their performance depends more on the underlying database engine, database design and hardware than on the campaign managers themselves. Similarly, third-party query, reporting and analysis tools are at least as good–and often better–than anything a campaign management vendor can build for itself. So, having given up control over most of the things that previously differentiated them, campaign management vendors now need new ways to compete.

Integrated suites meet this need nicely. They simplify implementation–one of buyers’ key concerns–while helping shift attention away from specific functional details. There is some irony to this development, since the integrated suites are in some ways as “closed” as the proprietary systems they initially displaced. Of course, most vendors don’t see it that way.

But back to the original question: are the campaign management functions in today’s leading systems so similar that buyers need not examine them in depth?

I think not.

Take query complexity. This is where the constraints of SQL bind most tightly: nearly any modern system can support a simple SQL query, and very few systems can do much else. The simple functions boil down to examining a single field to see whether its value is equal, greater than or less than a specified value or another field in the same record. Most systems can extend the comparison to include simple math operations such as addition and subtraction, and can also combine separate statements with “and” or “or” conditions (although it’s easy to make mistakes in setting up “or” logic.) But it’s hard to generate SQL to handle other situations, such as finding sets that do not contain a particular value (practical example: people who haven’t purchased a given product) or summarizing many-to-many relationships (example: multiple purchases linked to multiple returns). Most of today’s SQL-based campaign managers don’t even try to support such queries, relying instead on precalculation, hand-written SQL, or multiple steps in a segmentation flow. The most important exception, Decision Software Inc.’s TopDog (www.dsitopdog.com), actually extracts the data from the relational database, manipulates it without SQL, and then reinserts it.

Embedding precalculated values in a record is one way to overcome some SQL limitations. Today’s systems also vary considerably in this area. The most powerful allow users to design, save and reuse complicated formulas that incorporate multiple variables and can even look up values in related tables (for example, the median income of the customer’s Zip code). Systems also differ in whether the values can be recalculated automatically as a query is executed, and how much work it takes to store them permanently on the database.

Promotion complexity is also tightly bound to the nature of SQL. Most of today’s systems allow users to define hierarchical nested selections: that is, they apply a series of queries that use the records selected (or rejected) by one query as input to the next, in order to split the database into ever-finer segments. This is another way to create complex selections that would be difficult or impossible with a single SQL statement. The differences among existing systems arise in two areas. The first is that not all products actually allow this sort of nesting–some limit the number of levels of queries that are allowed. More subtly, systems also differ in whether they literally use the output of one query as input to the next, or instead generate independent queries that all run against the main database itself. So long as the independent queries reexecute the logic of all the preceding queries, either method will yield the same result. But the independent query approach requires a great deal more processing where large databases or many segments are involved. This can raise a scalability issue. Some products offer a choice of whether the queries are independent or not, although even within this group there are differences in how easy it is to set up the processes.

A related distinction is how systems handle segments that are separated in time–for example, where one message is sent thirty days after another. Here, the different queries pretty much have to run independently, since the latter message may depend on customer actions after the initial selection. The difference among systems lies in whether they create the underlying queries automatically, based on an interface that lets the user simply specify a time interval. Many do, but others require the user to build the time relationships into the queries themselves. This is difficult and error-prone, and makes it impossible to show the relationships on system reports.

Query and promotion complexity are areas where marketers’ needs vary widely–which is why leading systems have found it possible to treat them differently. Similarly, there are still large differences in the products’ ability to limit the number of promotions sent to an individual, in support for complex response analysis, and in administrative functions such as budgeting, project management and approvals.

On the other hand, some requirements are pretty much universal and these have been incorporated in every leading product. A good example are the Nth or random selections needed for standard testing procedures. These are available in all major systems even though random selects are quite difficult to create in standard SQL. Today’s systems also let the user design the underlying customer database rather than forcing data into a fixed schema, although many do have a standard schema for internal data such as campaign information. Most products also include job schedulers that can execute campaigns automatically over a period of time–another key requirement for successfully executing an advanced database marketing program.

In short, there are still substantial differences in the core campaign management features of today’s leading products–differences that can cause major problems if they are not compared to user requirements before a system is purchased. So while integrated suites offer real and intriguing advantages, it’s not yet safe to ignore the details of their embedded campaign managers.

Maybe it’s a sign of maturity or maybe it’s a sign of decadence, but today’s marketing software companies now seem more interested in acquiring other people’s products than building their own. The past year saw a spate of mergers and acquisitions among members of the industry. Some were traditional efforts to strengthen capabilities related to a company’s core offering. For example, Kana (www.kana.com), a provider of email response management software, purchased companies that do outbound email campaigns (Connectify), real-time Internet dialogs (Business Evolution, Inc.) and self-service Web-based support (NetDialog). Similarly, Sagent (www.sagent.com), a provider of tools to help load operational data into a warehouse or marketing system, acquired QMSoft, which has specialized technologies for data matching and enhancement. Roughly in this category–although perhaps more a consolidation in a shrinking segment–was the combination of Retail Target Marketing Systems with Experian’s AnalytiX group. Both provide traditional campaign management systems using proprietary database engines–an increasingly tough sale in a world committed to using standard technologies.

But other acquisitions had a bolder purpose: assembling a complete set of customer management capabilities. This group includes E.phiphany’s (www.epiphany.com) purchase of RightPoint, Broadbase’s (www.broadbase.com) purchase of Rubric, and, to a lesser extent, Exchange Applications’ (www.exapps.com) purchase of GBI and ClientLogic’s (www.clientlogic.com) purchase of MarketVision. The object of such transactions is to provide an instant infrastructure to firms with no existing customer management capabilities–transforming them from laggards to leaders in one simple step. Clueless to flawless, as it were. For marketers afraid of being left behind as their competitors do a better job of customer management, this is an irresistable proposition.

It is also the purest snake oil. Any experienced marketer or remotely honest vendor will tell you that the real work in becoming customer-centric is the organizational and process change. Delusions that simply buying the right product will solve the problem will be quickly shattered–though not until after the vendor has been paid.

This does not mean that integrated product suites lack value. The technoloyg of customer management is far from trivial, so selecting appropriate products does have a major impact on project success. This means it’s worth looking at just what the vendors of integrated suites have to offer.

The first step is to establish a framework for comparison. Nearly every marketing product claims to support “closed loop marketing”. The general impression conveyed by this term is of a system that executes marketing programs and captures the results so they can be evaluated and fed into improved future programs. But it turns out there are more loops in customer management than a three-mile roller coaster. Vendors naturally tend to define the kinds of loops that they can actually deliver. So it’s important to identify the components of the full customer management cycle in order to see which pieces any given vendor can support.

In broad terms, the cycle can be broken into five components. The first includes ‘front office’ systems that actually interact with customers: these are where transactions originate and where, eventually, marketing messages will be delivered. The second component includes the data extract and preparation tools that pull transactions from the front office systems and load them into the marketing database or data warehouse. For simplicity, assume this component also includes the database itself. The third component is the analysis tools used to assess the marketing data and guide marketing strategies. The fourth component is the campaign engine that assigns customers to groups and associates those groups with specific campaigns that embody the strategies. This component also generates outbound campaign messages. The final component is real-time interaction management, which feeds marketing decisions to the front office systems as customer interactions occur. Reactions to these decisions are captured by the front office systems and then fed back into the marketing database for analysis, thereby “closing the loop”.

It seems self-evident that a true “closed loop” system would contain all five components. But in fact there is a great divide in the marketing system world. Standing alone on one side are the front office systems, which are large, real-time transaction processing products run by operational departments like sales and customer service. These systems may have thousands of users in a large company; because of their size, they are where most of the money in customer management is spent. This is the turf of traditional “customer relationship management” (CRM) vendors like Siebel, Vantive and Clarify, and is being approached by other enterprise software vendors like Oracle and SAP. A related and somewhat overlapping set of vendors like Pivotal, Kana, Brightware and Silknet provide Internet-based front office products.

On the other side of the divide is everybody else: database, analysis, campaign and interaction management tools. These systems have much smaller numbers of users and (except for interaction managers) use mostly batch processing, which is better suited for most kinds of analytical work. Traditionally, these components have been provided by many small vendors who are specialists in their particular field.

With this picture in mind, the recent mergers and acquisitions come into clearer focus. E.piphany, Broadbase and Exchange Applications all are on the non-front-office side of the cycle, and so are the companies they acquired: E.piphany provides data transformation, analysis and some campaign management, to which RightPoint adds interaction management; Broadbase provides data transformation and analysis, to which Rubric adds campaign management; Exchange Applications provides campaign managment and some analysis, to which GBI adds outbound email. Thus, these acquisitions all represent an attempt to build multi-function integrated systems that can rise above single-function, stand-alone products that have traditionally dominated this sector. This strategy has several advantages: it leads to larger revenue streams; it makes the vendor attractive to companies that are new to customer management (still the vast majority); and it lets the vendor win against single-purpose products that are superior within their specialty. In fact, by helping deny revenues (and investment financing) to single-function products, the strategy has the welcome long-term effect of reducing competition and stifling the growth of firms that might eventually grow into full-spectrum rivals.

No strategy is guaranteed to succeed, and this one does has its drawbacks. The most obvious is that not all firms want to buy an integrated solution–and, in fact, the companies most likely to want a complete customer management solution are precisely the early adopters who already have partial solutions in place and therefore only need some incremental enhancements. These are a minority of the business world but may well constitute the majority of the market for the foreseeable future. Of course, the integrated product vendors are willing to sell their different modules separately. But these components are often less than best-of-breed and will be increasingly difficult to deploy independently as they become more tightly integrated with the rest of the vendor’s suite.

There is also the cost of expending technical and management resources on absorbing acquisitions rather than adding new system functions. This could lead the intergrated vendors to fall further behind specialists who create best-of-breed products in one area and build alliances with best-of-breed providers in others. This is the strategy adopted by Exchange Applications in setting up alliances with SAS and MarketSwitch for predictive modeling and with Microstrategy for reporting.

The integrated vendors also risk a backlash when buyers and investors realize that their “closed loop” products exclude the front office portion of the customer management cycle. Software buyers, who are used to vendor hyperbole, will presumably make this discovery between any money changes hands. But the investment community, entranced by visions of a multi-billion dollar CRM software market, may be less forgiving when discover most of the funds will be spent in the one area–the front office–that these vendors exclude.

Indeed, skillful handling of front office integration will probably be the key to long term success for the non-front-office vendors. Because the sheer size of the front office systems–in terms of complexity, number of users, operational impact and acquisition cost–they tend to be purchased on their own merits rather than as an adjunct to the other customer management products. This means it would be extremely difficult for a non-front-office vendor to successfully introduce its own front office products: in fact, this is so obvious that none of those vendors has even tried. Instead, they have created generic data exchange capabilities and, in many cases, have also built specific connectors to major CRM systems. Their problem is that the front office vendors themselves have their eyes on the non-front-office sector, and can either build or buy the components to provide the necessary services. So far, there have been no major acquisitions across the front-office/non-front-office divide. (ClientLogic’s purchase of campaign management vendor MarketVision is a near exception, but ClientLogic provides front office services rather than software.) But when the front office vendors start crossing the line, the non-front-office vendors will face a serious challenge.

Since interaction management represents the connection between front-office and non-front-office systems, it is the most likely point of contact between the two sets of vendors. Interaction management is also an uneasy fit with other non-front-office products, since it is inherently real-time rather than batch. This makes E.piphany’s acquisition of RightPoint–the most mature interaction management product–particularly intriguing. How E.piphany integrates RightPoint with its other offerings and manages its connections with front-office systems will be extremely important to its long term success.

Other products for interaction management are being developed mostly by traditional campaign management vendors (Exchange Applications, Harte-Hanks, Recognition Systems, Prime Response) or independent companies (Verbind, Trivida, Manna FrontMind, Black Pearl, DataSage). Whether the front office vendors acquire any of these, develop their own interaction managers, or leave interaction management to others bears close watching by both customers and investors in the marketing software industry.

* * *

The thrust of last month’s article was that mergers and acquisitions among marketing system vendors have generally expanded capabilities in either front-office or non-front-office areas, but rarely crossed the line between the two. (Front office systems, such as sales automation and call center systems, execute customer interactions; non-front-office systems, such as campaign managers and data analysis tools, help define strategies to guide those interactions.) The split is not surprising, since the two kinds of systems involve fundamentally different technologies and users.

But the split is also inevitably temporary. Once vendors fill out their capabilities in either area, they will want expand into the other to generate more revenues, block entry by potential competitors, and serve buyers who prefer one-stop shopping. While vendors in both areas will eventually want to expand, most of the revenue is on the front office side. This means the front office vendors are much more likely to become the buyers. Front-office systems are generally priced on the number of users, while non-front-office prices are usually determined by the size of the database. So this will be the Invasion of the Body Counters.

No sooner was last month’s article written than it became clear that the invasion had begun. In December, SAP (the largest provider of enterprise resource planning software, and a new contender in front office systems) announced an alliance with Recognition Systems Inc., developer of one of the most advanced and integrated campaign management/data analysis/email marketing tools. Ordinarily, alliances among software vendors are about as significant as two junior high school students announcing they are “going together”–they’ll talk on the phone a lot and attend a few parties together, and that’s about it. But SAP has historically avoided such relationships, so their decision to participate may actually indicate more significant intentions. If nothing else, the deal represents a recognition of the importance of non-front-office marketing functions as components of a complete customer management solution. Tactically, it also seems to be an attempt by SAP to outflank rivals who are well-entrenched within the front office itself.

Chief among these rivals is Siebel Systems, which countered SAP in January by purchasing Paragren Technologies, another provider of advanced campaign management software. Paragren relies on third-party systems for statistical analysis, but does have some interesting capabilities in data extraction and aggregation–other key functions missing in traditional front office systems. Since this is a full acquisition rather than an alliance, close integration of Paragren with other Siebel products seems more certain than integration of SAP with Recognition Systems.

While SAP and Siebel are the market leaders in the enterprise resource planning and front office segments respectively, Oracle is a major player in both. It has also moved to expand its non-front-office marketing systems, acquiring Darwin data analysis software from Thinking Machines and releasing significantly enhanced versions of its own campaign management products.

Of course, Oracle is primarily known for its database management software, and its purchase of Darwin also fits a pattern among database software vendors of expanding their analytical capabilities. For example, NCR last year added substantial analytical capabilities to its Teradata database, as well as announcing a major alliance with analytical software vendor MicroStrategy. Microsoft embedded a large set of analysis functions within its SQL Server software. And database vendor Informix purchased Ardent, a provider of data transformation tools.

As it happens, Ardent’s assets included a pretty good campaign manager called Customer Advisor, and NCR also released a suite of campaign management tools. But except for Oracle, none of these vendors seems especially interested in attacking the front office market itself. Their addition of analysis and campaign management functions is probably more related to their ambitions in the data warehouse market.

Naturally, front-office vendors also have their eyes on the Internet. Most have enhanced their front office capabilities to support email and Web interactions, although the prices of Internet software companies are so high that this is usually done through development or alliances rather than acquisition. Of course, the Internet companies themselves don’t have this problem, since they can buy each other using their own inflated stock as currency. This has let Internet front office vendors like Kana be active purchasers of other Internet software companies. Still, most of these acquisitions have stayed within the front office.

One recent exception is the acquisition of DataSage, developer of well-regarded data analysis and personalization software. The buyer was Vignette, whose Web site management software runs the Internet equivalent of the front office. Vignette’s primary competitor, personalized Web site expert Broadvision, has taken the opposite approach of building alliances with many non-front-office vendors rather than acquiring a single product of its own.

Whether alliances or acquisitions will prove the more successful strategy is so far unknown. Either way, the development of integrated product suites represents a major change for the marketing software industry. Instead of evaluating stand-alone products primarily on their technical merits, buyers now must first choose whether they want to accept the cost, time and risk of integrating separate products, or avoid those by purchasing a suite. This makes it considerably harder for stand-alone products to survive, both because their market is smaller–comprising only firms willing to undertake the necessary integration–and because they must lower their price to compensate buyers for the integration costs they could have avoided by sticking within a suite.

The existence of suites also changes the evaluation process itself. The larger set of functions makes it more work to compare products based strictly on functionality, and means it is less likely that one product will be superior to the others in all areas. In addition, functionality becomes less important because buyers must pay more attention to the vendor’s long-term viability: with so many more eggs in the same basket, buyers are at higher risk if the vendor fails. Both factors make it harder for new firms to compete even if they are technically superior or priced below the more established products. Of course, the desire to shift the terms of competition in their favor is exactly why vendors seek to create integrated solutions to begin with.

For software buyers, the significance of all this is that they cannot ignore industry mergers and acquisitions even though they might like to. At the minimum, buyers must consider whether any suite really does a good enough job to meet their current and foreseeable future needs–recognizing that once committed to a suite, it will be much harder to substitute a superior stand-alone application for any particular function. Of course, the suites themselves will also improve over time, but the need to retain close integration with many different components makes the suite vendors move more slowly than independent competitors. Buyers should also closely examine the actual degree of integration among suite components, particularly if some parts were acquired: components developed separately do not suddenly become compatible simply because ownership changes hands.

Above all, suite buyers should assess–and insist on–a suite’s ability to integrate easily with third-party tools. The marketing software industry is still evolving too quickly to assume that any single vendor will remain at the forefront in all areas. This makes it almost inevitable that the buyer will want to substitute an external product for some suite component at some point in time. Moreover, today’s marketing systems are increasingly expected to interact with other systems, both within the company and at outside suppliers and customers. This means the ability of a suite to work with other products is essential. Happily today’s technologies make such integration possible with little cost in performance, so it is not an unreasonable request.

The last two columns in this series have looked at ways to segment the universe of marketing-related systems. Although no fully satisfactory scheme has emerged, one distinction was present in nearly every attempt: batch vs.and real-time systems. The general argument was that the technologies needed for these two types of systems are so radically different that they need to be treated separately.

This proposition is worth closer examination–both to understand the nature of the technical differences, and to see how some systems manage to bridge the gap.

First, let’s get the definitions straight. Batch systems execute a sequence of steps without external inputs, while real-time systems wait for user input between steps in a transaction. Batch systems typically apply the same process–such as calculating a model score or assigning a customer segment–to many records in a single job, while real-time systems typically execute a process against a single record per job.

These differences in function result in different goals for system design. For a batch system, the key goal is to move through a large data set as efficiently as possible. The goal for a real-time system is to retrieve and update individual records with minimum delay.

Although batch systems usually process large numbers of records, they generally work with one record at a time: they read the record and its associated data, process it, store the outcome, and then repeat the process for the next record. Efficiency is determined primarily by the time it takes to assemble all the data needed to process each record. In a flat file system, this is done by either combining data from multiple sources into a single record before the process begins, or by sorting multiple files in the same sequence so the system can step through them in parallel without extensive searching. This sort of sequential processing is especially well suited to files stored on tapes rather than disk drives, since it allows the system to physically read the records in the sequence they appear on the tape. If the processing were not sequential, then the system would have to search for each set of records from one of the tape to the other. (Remember all those images of spinning tapes from TV shows and movies in the 1960′s and 70′s? That’s what was going on.)

In contrast, a relational database is explicitly designed not to place records in a specific sequence. Instead, relational systems rely on indexes to link the related data and typically load the data itself onto disk drives that can quickly access records that are not physically adjacent. Still, because sequential access is inherently more efficient than even the fastest disk drive, many of the largest-volume batch systems create an ordered extract that is then processed like a flat file.

Relational systems also often improve efficiency by “denormalizing” the data, which means storing the same piece of information in more than one record. This violates a cardinal rule of relational database design, which says each item should be stored only once. The rule exists to ensure data consistency and speed updates. But violating it will reduce the number of tables that must be searched and read to process a record. This can yield major performance gains.

Batch systems can get away with denormalization and sequential processing because they are not subject to the same constraints as real-time systems. Most real-time systems don’t know which record will be needed next, because they are reacting to unpredictable events such as which customer will place an order or call for service. Therefore the real-time systems need search mechanisms like indexes on account numbers, which allow them to find any particular record quickly. By contrast, a batch system will eventually process all records in its set, so has no particular need to locate a specific record first. Real-time systems also must be kept internally consistent at all times, since two transactions relating to the same account might occur almost simultaneously, and different kinds of transactions might occur in different sequences. This makes it much more dangerous for real-time systems to violate the relational principal of “normalization”–storing each piece of information only once–than for batch systems, which exist in a much more controlled environment. Similarly, real-time systems are also more focused on the update speed that normalized designs provide.

So, to oversimplify a bit, batch systems use sequential processing and denormalized data structures (few tables with some redundant data), while real-time systems use indexes, random access and normalized structures (many tables with no redundant data). While it’s possible for one system to do both, most software is optimized for one or the other. This is why the distinction is so fundamental when attempting to classify different marketing products.

Specifically, traditional data warehouses and database marketing systems tend to use batch processing techniques–after all, most queries are looking for patterns or segments in the entire database, not picking out a single customer or account. By contrast, front-office systems for customer service, sales automation or contact management are real-time systems that must be designed to work with one customer at a time.

The problem, of course, is that today’s goal is to merge the back-end marketing database with the front-office customer contact system. This lets users define customer strategies in the back-end system–which has the rich history data and analytical capabilities–and execute the strategies in the front-office system during the real-time interactions. So designers are being asked to make one system handle both batch and real-time processing.

As with most computer processing challenges, there are two basic solutions: brute force and elegant design. Given the continued drop in hardware costs, brute force is often the best approach. But in some situations, elegant design is still worth the effort.

In dealing with real-time marketing systems, the classic application of brute force is parallel processing. This involves systems that split a single batch job into many smaller jobs and run them all simultaneously. IBM’s SP2 and NCR’s Teradata are the most common examples of massively parallel systems, although other vendors have products as well.

Massively parallel systems do have the ability to give high performance on both batch and real-time jobs. But the hardware is expensive and developers must usually tune the application software and data structure for optimum performance.

This tuning is costly and time-consuming, which is bad enough. But it also means that the resulting system may perform poorly when faced with unanticipated demands. For example, one common tactic in parallel system design is to store data from different date ranges on separate hard drives (each served by its own processor). This works great when queries look across all date ranges, since the different processors can work on the different date ranges simultaneously. But if queries suddenly focus on a single date range, the system will slow considerably because only one processor can access the necessary data. (Reality is a bit less grim, since parallel systems can usually give several processors access to the same data if necessary. But performance will still suffer.)

A newer brute force approach involves “main memory” databases, which essentially move the underlying data from a disk drive into high speed, random access memory. Specialized database management systems that do this include TimesTen (www.timesten.com) and Angara Data Server (www.angara.com). These systems can access records ten to twenty times faster than if the data were stored on a disk drive; they can also employ specialized indexes that reduce performance impact of bringing together related records from many different tables. The most important current application of this technology is managing Internet interactions, where systems may need to access huge volumes of data in real time. But the fast access provided by the main memory systems allows them to complete batch processes extremely quickly as well.

For companies that are unable or unwilling to apply brute force solutions, the alternative is a system design based on conventional technology. Since the same conventional data tables generally cannot provide adequate performance for both real-time and batch tasks, this usually involves maintaining separate data tables for the two types of applications, and somehow synchronizing them. The simplest approach is to first load all data into a conventional marketing database–structured for batch processing–and periodically create extracts that are structured for access by real-time systems or feed data into the real-time systems’ own tables. The problem with this method is that batch processes are used to update the conventional database and to generate the extracts. This means the marketing system cannot feed adjusted information as a transaction occurs. So the marketing feed itself is something less than real-time.

A slightly more sophisticated approach is to update the table that supports the real-time systems at the same time that the main marketing database is updated. This avoids the lag due to batch extracts, but still must wait for the batch updates of the main database. The only way to avoid this second lag is to update the real-time table directly, rather than filtering data through the main marketing system first. Some systems–particularly those designed for Internet marketing–do maintain a profile database that is updated in real time in this fashion. In addition to simply capturing the new transaction, such a system might recalculate derived values such as cumulative purchases and model scores, and use the adjusted values in managing the interaction. The new data would be periodically added to the main marketing database during its regular batch update. This sort of synchronization is about the best that can be done with conventional technology.

As marketers continue to integrate real-time front-office systems with batch-oriented marketing databases, vendors will face increasing pressure to combine batch and real-time processing in a single system. As we’ve seen, this is a difficult task using today’s standard (relational) technologies. Buyers looking for an integrated system should look carefully at each vendor’s approach to this challenge, to ensure the system they purchase will meet both current and future needs.

So much software is offered today to marketers that just trying to evaluate each new product is more than a full time job. But important as it is to understand the strengths and weaknesses of individual products, it’s also necessary occasionally to step back and understand how products relate to each other. This helps to ensure whatever product you are thinking about buying today will fit into the larger structure that will evolve over time. It also helps answer the important question of whether the product’s developer is likely to survive and prosper in the future.

Most computer industry analyses use a two dimensional matrix. This limits the amount of information that can be conveyed about any individual product, but it has the virtue of being easy to understand. Let’s just accept the two dimensional limit and consider what those dimensions should be.

The answer depends on what you’re trying to accomplish. Take the standard matrix used by a well-known IT advisory firm, comparing company “vision” with ability to execute that vision. Those are pretty useful measures if you’re considering investing in a company, either financially or as a buyer of its products. The vision axis gives some idea of whether a company’s products are likely to meet the long term functional requirements of a sophisticated user, while the execution axis hints at both financial stability and resources available to help less sophisticated users with implementation. In combination, the two measures are terrific at annointing “leaders” in a given category–an item of considerable interest to certain buyers and great promotional value to the vendors themselves.

Unfortunately, both measures are also highly subjective. In particular, you may not agree with an analyst’s definition of what constitutes a quality “vision”. More dangerously, this sort of competitive ranking implies there is a single “best” product for all users. In reality, user’s needs vary widely and the right product for one user may be totally inappropriate for another.

Let’s assume you want concrete help in selecting a marketing system. Now you want dimensions that more specifically indicate the functions provided by a product and differentiate similar products from each other. Of course, no two dimensions can capture all the issues. Still, some interesting efforts have been made.

One approach distinguishes analytical vs. execution functions–based on the observation that these have been done by separate systems in the past, but today some products offer both. Purely analytical products would include model building tools like SAS and ASA ModelMax, while pure execution tools would be telemarketing and list generation systems. Hybrid products would include Recognition Systems Protagona and Unica Impact!, which have tightly integrated modeling and campaign management. This method has several advantages: it distinguishes integrated from non-integrated products, helps determine which systems would be complementary rather than overlapping, and lets users choose the quality of analytical and execution functions they require.

But this method doesn’t indicate which channels a system supports. This means that an email broadcasting system and inbound call center application could occupy the same spot on the matrix–even though the two are utterly different products. It also means that a system supporting multiple channels looks the same as one supporting a single channel. Either way, the matrix is missing a key distinction.

It’s possible to imagine a matrix where one side represents the channels served by a product, perhaps arranging the different channels in a logical sequence such as cost per contact or speed of execution. The other dimension would then indicate how well the system supported each channel. The result would be a visual profile of the strengths and weaknesses of each product–a pretty useful thing for some purposes. But this approach displays each product as an irregular blob with multiple data points, which means the simplicity of the two dimensional matrix is lost. When more than a few products are plotted, the results quickly become unwieldy. If you want this type of detail, it is better to use a table with checkmarks or scores for each product’s capabilities in each channel or function.

A simpler approach that does fit within two dimensions would arrange systems based on the number of channels (or other functions) they support. The second dimension could indicate quality–that is, how well the system supports the channels it services. Like the earlier analystical vs. execution matrix, this breadth vs. quality approach has the problem of placing very different systems next to each other. But by putting multi-purpose systems at one end of the matrix and specialized systems at the other, it does distinguish two of the most common vendor strategies: providing a large number of integrated functions vs. doing a single function better than anyone else. This makes it very helpful for buyers who prefer either an integrated package or to assemble their own system from “best of breed” components. Such a matrix would also identify any integrated vendor with high quality components–since it’s at least theoretically possible for an integrated system to be good at everything. Of course, where multiple components are involved, the quality measure would need to be some sort of average, and thus require more detailed explanation to assess quality of individual functions.

As an alternative to quality, some analyses look at the breadth of vendor offerings–specifically, indicating whether a vendor provides software only, software plus supporting services such as application hosting or implementation, or services only. Such a breadth of function vs. breadth of service matrix is very helpful in further distinguishing different vendors’ strategies and identifying vendors who match a particular buyer’s needs.

A different approach focuses on the characteristics of systems that support different marketing functions–that is, distinguishing conventional campaign management from email campaigns, customer service systems from Web-based message delivery, and so on. It is possible to array these different systems based on response cycle (from batch to real-time) as one dimension and interaction complexity (from simple rules to complex customer strategies) as the other. Such a matrix would range from simple list generators (batch processing, no rules) in the lower left to online interaction managers (real-time reaction, long term strategies) at the upper right. Other types of systems would have different combinations: for example, recommendation engines like NetPerceptions give real-time results but rarely look beyond the goals of the current interaction (upper left); conventional campaign management software supports long term strategies with batch processing (lower right).

This matrix offers some interesting insights, since very different technologies are needed for batch vs. real-time processing and for simple rules vs. long-term strategies. In particular, it suggests that vendors claiming to straddle more than one category need to be questioned closely about exactly how they do it. It also raises questions about vendors who started in the simple rule segment but are now attempting to support more complicated strategies. For example, many of today’s “customer relationship management” products started with sales automation or call center systems (simple rules, real-time interaction). Based on where this puts them on the matrix, should be no surprise that campaign management is the weakest feature of their products. Conversely, the matrix correctly predicts that conventional campaign management vendors (batch processing, complex strategies) will have difficulty adapting their systems to handle real-time interaction.

By now it should be clear that no pair of dimensions can fully describe the relationships among different marketing software products. But it should also be clear that a carefully chosen matrix can highlight issues that are important in a particular situation. As always, the burden is on the user to understand her needs and structure an analysis that addresses them correctly.

* * *

Last month’s column described the impossibility of capturing all the significant differences among marketing software products in a single two-dimensional matrix. It’s still impossible, but the reality is that buyers and vendors do need a way to make sense of the different systems. So let’s look at yet another matrix that at least manages to distinguish the main classes of products and how they relate to each other.

The horizontal dimension of this matrix measures reaction cycle–ranging from batch processes on the left to real-time interactions on the right. Batch processes sometimes run every few minutes, but in marketing systems they usually run no more often than daily, and many times just weekly or monthly. Whatever the interval, the important point is the systems respond too slowly to influence whatever transaction is taking place. By contrast, true real-time systems react immediately, in a few seconds or less, and therefore can participate in an on-going interaction. Common real-time systems are telemarketing scripts that tell an agent what to say next and Internet servers that return a page in response to a mouse click. Between batch and real-time are systems that react promptly but not immediately, such as customer support products that reply with an email or fax within a few minutes of a customer inquiry.

Loyal readers will remember that last month’s column also proposed a matrix with a reaction cycle dimension. The other dimension of that matrix had to do with interaction complexity, which roughly corresponds to the sophistication of the contact management strategy a system can execute. That was a pretty useful matrix, but it lumped together fundamentally different systems like low-tech call centers and high-tech collaborative filtering products (which both belong to the real-time, simple strategy group). And that matrix totally excluded modeling and analysis systems, which don’t manage interactions at all.

The second dimension of the new matrix measures analytical sophistication, which ranges from automated modeling systems (high) to user-specified segmentation schemes (low). Assume the high sophistication is at the top and low sophistication is toward the bottom. In between would be rule-based systems that can make sophisticated decisions but rely heavily on user input to specify the underlying rules.

It also turns out that analytical sophistication generally correlates inversely with execution capabilities–that is, systems built to execute marketing programs tend to have limited analytical power, while those with high analytical power rarely do much execution. There are some exceptions to this rule, but they are intriguing enough that it’s actually useful to have to deal with them separately.

So let’s look at how this new matrix lays out. It proposes two major distinctions: batch vs. real-time and analytical vs. execution. The four possible combinations do indeed correspond to familiar classes of systems:

- in the “batch analytical” corner (upper left) are the traditional advanced analysis tools, including conventional statistical packages like SAS and SPSS, neural network software like Trajecta and Advanced Software Applications, and multidimensional analysis tools like Hyperion Essbase and Oracle Express. In fact, sophisticated analysis has always required batch processing, which has become an increasing problem for marketers who want to reduce cycle times. The best these traditional tools can do is to build their models in batch, but score individual records in real-time or near real time.

- this leads to the “real-time analytical” corner (upper right), which today is populated by recommendation engines like Net Perceptions and Andromedia Likeminds, and by interaction managers like RightPoint, Manna FrontMind and Trivida. These products both predict a specific individual’s actions in real time and actually adjust the underlying models as new behavior is recorded. Like conventional modeling tools, the real-time systems have very little execution capability of their own–they only feed their predictions to other systems that manage the actual customer contacts.

- specifically, they feed “real-time execution” systems (lower right). These include conventional call center and contact management products like Siebel and Clarify, as well as personalized Web site systems like Broadvision and Vignette. Although there are major technical differences between conventional and Web-based execution systems, from a marketer’s standpoint they are just different ways to deliver the same contact strategy. So it does make sense for the matrix to group them together. And, regardless of the technical differences, vendors are striving to integrate the two sets of products–so there will soon be no choice but to treat them as one.

- the final corner holds “batch execution” products (lower left), which perfectly describes old-style campaign management software like Experian AnalytiX and MegaPlex FastCount. These products use proprietary database engines that are loaded in batch and used primarily for batch selections of mailing and telemarketing lists.

So far so good–the four corners of the matrix describe distinct and important classes of systems. In fact, people who care about such things might notice that the four corners correspond to the major components of a standard enterprise architecture: operational systems (real-time execution), data warehouse (batch analytical), campaign management (batch execution) and interaction management (real-time analytical). Kinda neat, huh?

But what about the spaces between the corners? Along the execution edge of the reaction cycle dimension (the bottom of the matrix), today’s advanced campaign managers like Exchange Applications ValEx and Prime Vantage might be considered “near batch” products: they mostly use batch loads and selections, but have schedulers and other functions that let them respond to events fairly quickly. They have also been integrated to some degree with outbound email and email responses, also pulling them slightly in the real-time direction. Further along that edge are email campaign managers like Responsys and RevNet, which are used primarily to broadcast batch-selected emails but can also capture email replies and issue a predefined response. Still closer to real time execution are email customer service systems like Acuity and Brightware, which can provide unstructured responses to email inquiries in near real time. Like the call center and Web site systems mentioned earlier, these products are increasingly being expanded to handle additional media, including true real time interactions such as telephone calls and live Internet chat. Nestled between email customer service and real time interactions are the various “marketing automation” products like Imparto and MarketFirst. These can handle both near-real-time response via email and true real-time interactions via personalized Web pages.

Above the pure execution layer lies the middle ground between execution and pure analysis. This is occupied by rule-based systems that rely on people to define a set of policies, but then can combine and apply them independently. Systems including Harte-Hanks Allink Agent and NCR’s CRM trio of Marketing Agent, InterRelate+ and Relationship Optimizer can scan for significant operational transactions in near real time and apply rules to determine how to respond. Black Pearl’s Knowledge Broker, along with RightPoint, can do the same thing in true real time.

Also in this middle ground are the exceptional products that offer both analysis and execution. (I have somewhat arbitrarily placed them above the rule-based layer.) In pure batch processing, Unica Impact! offers a powerful campaign manager plus extensive model building. E.piphany and Broadbase also combine analysis and selection capabilities, although they are less capable in both areas. In the near batch group, Recognition Systems Protagona offers its own integrated modeling, an excellent campaign manager, and a respectable degree of email interaction. Web traffic analysis–a batch or near-batch pure analytical application in products like Accrue and net.Genesis–is also combined with execution by several systems including iLux, GuestTrack and Personify.

As the list of exceptions suggests, today’s relatively neat distinctions can be expected to fray over time, as vendors expand their products to encompass functions in more categories. The matrix has other flaws as well: it doesn’t indicate which channels a product supports, doesn’t identify vendor services such as application hosting, and says little about quality. But it does manage to encompass most of the systems marketers worry about today, and hopefully that is useful enough.

* * *

Some phrases have charisma and others simply don’t. Successful terms like “customer relationship management”, “knowledge management”, “data warehousing”, and “data mining” all somehow sound important, exciting and complicated enough to justify large sums of money and conferences in desirable locations. Other terms, like “cost-benefit analysis”, just don’t make the cut.

“Optimization” will never be a really hot buzzword: it sounds too dry, too limited to wringing the last bit of value from a well-worn set of options. This is emotionally unappealing: people want to blaze a new trail through the wilderness, not cut two minutes from their trip to the grocery store. It is also a dubious business strategy: with the rapid change and new opportunities of today’s environment, there truly are new wildernesses to explore. So fine-tuning an existing process just doesn’t seem all that important.

Still, while optimization will never attract stadiums of screaming fans, it does have its own followers–particularly among the analytically minded, and in industries that are relatively stable. In fact, the term is popping up with surprising frequency in vendor presentations these days. Unfortunately, different vendors use it in different ways–a common enough situation, but one that will further contribute to the term’s ultimate lack of utility.

In the hopes of salvaging some value from this soon-to-be-overused word, let’s take a closer look at what it can mean.

First stop, dictionary. My ancient one defines “optimize” as “to be optimistic”, but then gets around to today’s more common meaning of “to make as effective, perfect or useful as possible”. The key here is “as possible”: because what optimization systems truly do is manage sets of constraints. The focus on constraints is inherently pessimistic, and part of why “optimization” is psychologically unappealing. But, more important, it also gives hint of how to classify optimization systems: by looking at the type of constraints that they manage. The major distinction might be called tactical vs. strategic optimization.

Tactical optimization manages constraints related to a single decision. This kind of optimization has been around for a long time–it is as simple as finding the exact mailing quantity that will yield the highest profit on a list of names ranked by expected response rate. Today, any decent predictive modeling software provides this capability, usually in the form of a “gains chart” that shows the expected costs, revenues, profits, and response quantity from mailing to different depths in the ranked file. The better implementations–such as MarketSwitch Corporation’s Targeting Optimizer (www.marketswitch.com) and Group 1 Software/Unica Model 1 Campaign Optimizer (www.g1.com or www.unica-usa.com)–provide a slick graphical display that shows how these metrics change with different mail quantities, and even tell the user what quantity will meet specific constraints such as a fixed promotion budget or target number of new customers.

MarketSwitch’s Cross-Selling Optimizer takes this a step further including multiple offers subject to their own constraints–such as a maximum promotion quantity or minimum sales target per offer. This is in addition to customer-level constraints such as a maximum number of offers or minimum profit per name. The output is a plan that assigns treatments to each customer in a way that is expected to yield the best over-all result.

But whether the optimization involves one offer or many, what makes these approaches “tactical” is that they consider only the results of promotion at hand. The result is typically measured in immediate profit or return on investment, although it could also incorporate future values such as lifetime purchases from a new customer. While any sensible marketer realizes the future value is determined in part by future decisions, tactical optimization systems themselves do not attempt to measure or manage the future alternatives.

Strategic optimization does exactly this. That is, it looks at a sequence of future decisions and outcomes, and attempts to find policies that will yield the highest long-term value. This is a much more ambitious undertaking than tactical optimization, and probably needs a more exciting buzzword to capture its importance. Of course, one could argue that “customer relationship management” already does this quite nicely.

Semantics aside, the importance of strategic optimization is that it offers the ability to change the long-term value of an existing customer relationship. This involves two major tasks: figuring out what the optimal policies are, and finding ways to implement them. Today, these tasks are handled by separate systems–although there is no particular reason a single system to do both might not appear in the future.

Developing optimal policies is the greater challenge, because it involves true creativity: thinking up a new product, or type of offer, or service policy. Of course, no computer system can really do this today; the problem is simply too unstructured. (Some advocates of artificial intelligence may disagree, but that’s another discussion.) Still, a computer system can report on the results of past policies, predict what will happen if the same policies are applied in the future, and perhaps even estimate the results of combining them in new ways. This involves lots of model building and simulation, so if the number of options to consider or events to predict increases beyond a fairly limited point, the volume of work becomes overwhelming for even the largest computers. This is one reason that strategic optimization has so far been applied primarily in the credit card industry, where there are a limited number of key options (interest rate, credit limit, annual fee, grace period), relatively few key events (activation, balance maintenance, payment, renewal), and lots of customers to provide data and amplify the value of any improvements. Credit cards are also a fairly stable industry with lots of analytical people in control.

The simulation inherent in strategic optimization also lets users examine the risk posed by different sets of policies–say if interest rates rise or bankruptcies increase. While this simulation could also be run without optimization, it’s nice to have both in the same system.

But even in the credit card industry, compromises are necessary to make strategic optimization practical. Trajecta (www.trajecta.com), which seems to have the most complete approach to this problem, limits its analysis to a handful of key variables and combines detailed modeling of near-term events with simpler forecasts of long-term behavior. Both shortcuts are justifiable: a few variables do account for most differences in behavior, and detailed long-term simulations are unlikely to be more accurate than simpler forecasts. But the shortcuts also mean that other tools would be needed to deal with more complicated industries or to make optimal decisions about non-key variables.

This last point is particularly sticky. It’s easy enough to argue that a handful of key decisions account for most of your business profit, and maybe you can even prove it with statistics. But try explaining this to the CEO who just spent $20 million for a new call center precisely because it was able to personalize every customer interaction. Chances are pretty good that she’ll want to treat different people differently, whether or not the optimization system can tell her how.

In fact, the call center rules will probably be defined the old fashioned way: by human beings making their best guess about what policies make sense, and then (hopefully) watching the results to improve the rules over time. This is the realm of the other strategic optimization systems, which do implementation.

The classic rule-implementing optimization systems also originated in the credit card industry: venerable products like Fair-Isaac TRIAD (www.fairisaac.com) and AMS Strata (www.amsinc.com), and the more recent HNC Capstone Strategy Manager (www.hnc.com) and Trajecta Decision Optimizer. All let managers define strategies comprising rules for key decision points, assign customers to different strategies, execute the strategies and evaluate the results. TRIAD and Strata, with roots stretching back more than a decade, have also been adopted in other financial services and telecommunications. These systems are usually integrated with operational processes such as billing so the appropriate decisions can be made and executed during the normal course of business. Optimization evolves over time as managers set up champion/challenger tests that assign customers to alternative strategies, compare the results and pick the winners. Although these systems could also be adapted to selecting names for outbound communications, like a conventional direct mail campaign manager, this is not the usual application.

Recently, however, there has been some movement toward outbound optimization. Recognition Systems Protagona (previously ideas Solution; www.recsys.com) and NCR Relationship Optimizer (www.ncr.com) includes extensive features to manage constraints such as maximum number of contacts or promotion expenses per customer over a time period. Protagona even takes a stab at balancing revenue received from a customer with value provided to the customer–a particularly knotty problem that most vendors more or less ignore by assuming the user will develop a long-term measure of value that encompasses both. Both systems also accommodate limits on marketing resources such as call center capacity. Relationship Optimizer can automatically track the load on marketing resources as responses come in, and shift lower-priority messages to alternate channels when necessary. Although lead management and call center systems have provided similar cascading functions for years, they are unusual in a campaign management system.

Or is there really a distinction between “outbound optimization” systems like Relationship Optimizer and an advanced front office system like a Siebel call center? True, both can implement customer-tailored business policies. But the ability to embed and analyze policies in campaigns and strategies is very limited in standard front office systems: anyone who wanted to develop true optimization would find it difficult at best. This may change over time as the front office vendors strive to make their products live up to the optimization claims inherent in the concept of customer relationship management. On the other hand, tools like Protagona and Relationship Optimizer most definitely do not provide the operational functions of a call center, sales automation or Internet response management product. That is, they don’t capture customer data or execute transactions. Like all strategy implementation systems, they are decision engines that tell other systems what to do–whether it is a batch process processing credit card statements, an on-line queue of messages to display at a bank teller station, or a real-time response to a customer action. Even if the front office vendors were to expand their strategy management capabilities, it seems unlikely that they would extend beyond messages delivered through their own customer interaction tools. So independent strategy implementation tools will probably remain necessary to truly coordinate–and optimize–all decisions regarding each customer.

But I still don’t think they’ll call it optimization.

Telecommunications analysts sometimes refer to POTS–Plain Old Telephone Service, in contrast to the fancy new services now available. In the same way, it’s useful to distinguish Plain Old Campaign Management–the ability to extract file segments, promote them, and analyze the results–from the advanced functions that campaign management software vendors are now adding to their products. Until a couple of years ago, there were so few products that just having Plain Old Campaign Management was enough to make a system interesting. But today, so much campaign management software is available that a vendor needs something more to stand out from the crowd.

But what to add? At one time, industry observers (including this one) spoke confidently of Third Generation campaign management, which would control customer interactions as they occurred. But few systems have actually appeared with that capability, and only a handful of companies have implemented it. So it appears that everyone will not evolve in that direction.

Instead, campaign management vendors have chosen to add many different features. Here are some of them.

- integrated modeling. Predictive modeling has long been a key element in database marketing. Many older products have offered loose integration with third-party modeling software, typically through interfaces that make it easy to extract data for modeling and to import scores or scoring formulas. But newer systems including Recognition Systems Protagona (previously ideas Solution; www.recsys.com) and Unica Impact (www.unica-usa.com) provide model-building capabilities within the campaign management software itself. This allows non-technical users to use model scores in their campaigns with a minimum of effort–although a significant level of skill is still needed to make sure the systems are applied properly. It will be interesting to find how widely these modeling functions are actually used: in the past, vendors with integrated modeling modules have reported that while many prospects asked if they were available, few ended up purchasing them. Everyone else either continues to rely on traditional models generated by professional statisticians, or does without.

- multidimensional analysis. Like modeling, data analysis has always been an important part of campaign management. Early vendors built their own analysis tools, since they used proprietary database engines that could not be read by anyone else. But once campaign management systems moved to standard relational databases, analysis was typically handled in third-party reporting software such as Crystal Reports or Business Objects. As multidimensional analysis tools like MicroStrategies DSS Agent, Oracle Express and Hyperion Essbase gained popularity, campaign management vendors added links to those products as well. (Multidimensional analysis uses data organized into common categories such as time, product, geography or customer segment; it is widely considered the most effective way to let non-technical users do detailed data analysis. It is also often called On Line Analytical Processing, or OLAP.) But integration with the third-party multidimensional software had its limits: segments identified on a multidimensional report usually could not be imported directly as a promotion selection, and changes in the marketing database were not automatically reflected in the multidimensional database. Today, E.piphany (www.epiphany.com) does campaign management directly on a multidimensional data structure, still using a standard relational database engine. This simplifies administration and allows tight integration between the multidimensional analysis and standard campaign management functions. E.piphany also provides extensive functions for transforming operational data into the marketing database–another function that was built into the old proprietary systems, but is usually handled today by third-party or custom software.

- loyalty programs: traditionally, specialized software has been used to run marketing programs that issue rewards for customer purchases. Like conventional campaign managers, these systems work with customer transaction data. But they also need to credit points as they are earned, look up balances, and issue awards. These imply data structures and interface screens that allow direct access by data entry staff, which are significantly different from the structures and interfaces used in a conventional campaign management system. Still, campaign management products including RMS MarketEXPERT (www.marketexpert.com) and STS Open MarketWorks (www.stssystems.com)–both developed for retailers–have modules to provide these functions. MarketEXPERT also has a “marketbasket” module to identify products that are typically purchased together. This is another task that is usually performed by specialized software.

- contact management. By definition, contact management systems allow users to schedule, execute and record one-on-one contacts such as telephone calls or in-store conversations. Like loyalty functions, these require transaction processing technologies that are foreign to standard campaign management architectures. But vendors including AIMS Software (www.aims-software.com) and MarketVision (www.marketv.com) have developed contact management modules that provide these abilities despite the technical challenge.

- profitability analysis. Bank marketers have a particularly difficult time measuring product and customer profitability. One reason is that much of their cost is related to the services consumed by a customer–for example, a customer who visits a branch three times a week might cost ten times as much to serve as a customer who conducts a single ATM transaction. Gathering this data from operational systems can be a major development project, which many banks extend to include their own profitability measurement system. Other banks use third-party profitability software or rely on an external service. But bank marketing systems including Centrax Marquis (www.marquismcif.com) and Acxiom Solvitur CIMS (www.acxiom.com) also offer detailed profitability reporting when the necessary data is available. The Acxiom product, which uses technology from data analysis software vendor Information Advantage, also extends beyond traditional campaign management to provide users with personalized “portals” to access a variety of marketing-related messages, data and reports.

- personalized e-mail. Traditional campaign management systems generate lists of customers to be sent promotions. These lists might include information for personalized letters. But most systems are not built to print the letters themselves because such work is usually done at external vendors with specialized equipment such as high speed printers. Personalized e-mail could be handled the same way, by having the marketing system just provide information to feed other systems. But many companies manage their e-mail internally, so it is less likely that an external vendor will be needed to produce the final electronic communication. Campaign management vendors including Ceres (www.ceresios.com), Prime Response (www.prime-response.com) and Recognition Systems (www.recsys.com) have developed products to generate the personalized e-mails directly. Prime and Recognition can also help generate personalized Web pages, although neither can quite do the task by itself.

- other functions. Vendors have extended campaign management in still other directions, including work flow (tracking tasks such as copy writing and budget approvals), promotion calendars (showing all promotions planned during a specified period), optimization (picking the best promotion for an individual), integrated mapping, automated job execution and, of course, real-time interaction management. Some of these functions are quite popular–for example, nearly every system today has a scheduler to execute campaigns automatically. But most of the additional functions are needed by only a small set of customers, depending on their industry, sophistication, or business strategy. So it seems that while Plain Old Campaign Management is no longer enough, there are many alternate approaches that may lead to vendor–and client–success.