David Raab Article Archive

One of the major challenges in building a customer-centric database has always been matching records from different systems that refer to the same customer. The necessary technologies–name and address standardization, postal costing, matching and householding–are fairly mature and quite familiar to firms with a long history of database marketing. But they are new to firms without this background and require a level of implementation skill that most will not have available in-house. Nor it is easy to add such skills, because experienced technicians are rare and because even if you hired one, they wouldn’t get enough practice at most companies to maintain their expertise. As a result, many firms either hire consultants to set up their systems or farm out the whole process to a service bureau.

Both of these approaches have drawbacks. Consultants can be expensive–though they are a bargain compared with the cost of failure–and may not train in-house staff to run the system after they leave. Service bureaus are even more expensive, and typically require days or even weeks to complete an update cycle. Perhaps worst in today’s time-pressed environment, it can take months to set up, test and refine a satisfactory in-house or bureau-based matching process.

Over the past few years, a new alternative has emerged. This involves using an external matching service on a real-time basis: that is, sending each new record to be processed as it is entered, and then loading the result directly into an in-house system. Vendors including Acxiom, Experian, Sagent, iMarket and Dun & Bradstreet all offer some version of this approach. While not necessarily cheaper than conventional methods, it offers near-immediate deployment, better quality than most firms could achieve themselves, and the option to simultaneously enhance the new record with external descriptive data.

External matching uses basically the same technology as conventional matching–that is, records are parsed into name and address components, standardized against postal and other reference tables, and then matched against other standardized records. Parsing and standardizing a single record takes just a fraction of a second, and poses no particular technical challenge. But matching has traditionally been a batch process that involved sorting the entire set of records to bring similar records together and then comparing neighboring records to each other. This won’t work with a real-time process, because each new record would have to be inserted into the sorted file as it was added, so future records could match against it. Moreover, maintaining a separate file for each client would be expensive and involve a time-consuming setup process.

Instead, external matching systems use a standard reference table of all individuals and households or businesses in the U.S. (or whatever market is being served). Such files, compiled from a variety of sources, are readily available to large service bureaus. Once an incoming record is standardized, it can be matched directly against this file regardless of what other records are in a particular company’s existing database and without any need to insert new records (except the handful that don’t match the reference table itself–which may or may not be worth adding). Each record in the reference file is assigned a unique, permanent ID. This means that when a match is found, the system merely appends this ID to the new record and returns it to the company, where it is easy enough to check whether a record with that same ID is already present in the company’s database. Thus, external real-time matching can be highly efficient from both a processing and storage point of view.

The reference table also lets the vendor pre-associate demographics and other enhancement data with each record, making it easy to return that data with the matched record itself. Similarly, the vendor can predetermine which individuals belong to the same household, so no on-the-fly processing is needed to return a household ID. The vendor can also use the permanent ID to link old and new addresses for the same individual, so if an old address is presented it can return the new address as well. Because the same reference table is used by many different clients, the vendor can afford to update it frequently, thereby providing even low-volume marketers with the most current information.

Unfortunately, assigning a standard ID to each indvidual also raises significant privacy issues. Since all match requests are processed against the same reference table, it would be easy enough for the vendor to keep track of which names have been presented by which sources–creating an unprecedentedly complete activity profile. Achieving the same result with conventional technology requires a massive matching job of many individual lists against each other, something that only very large marketers, compilers or name rental cooperatives can afford. The same technology would also let marketers subscribe to information about specified individuals–that is, be notified when a particular individual moves, makes a significant purchase, is reported to have an income change, or simply shows up on somebody else’s list. Again, conventional technology could only do this through periodic batch processes rather than continously.

The vendors of these services have policies that are designed to prevent what they consider privacy abuses, although some consumers might have different standards. The vendors also have strong economic motives to ensure that list owners do not use the standard IDs to share data without the vendor’s participation, and that list owners can use their services without fear their information will be used by others. So there are some safeguards built into the system. Whether these are ultimately deemed adequate remains to be seen.

In short, external matching provides an attractive alternative to conventional techniques in many situations. Marketers should be sensitive to privacy issues but take advantage when appropriate of its significant benefits.

* * *

David M. Raab is a Principal at Raab Associates Inc., a consultancy specializing in marketing technology and analytics. He can be reached at draab@raabassociates.com.