David Raab Article Archive

Every year around this time, Raab Associates updates its Guide to Customer Matching Systems. After changes for each vendor have been posted, we have a rare opportunity to step back and look for larger patterns. Here’s what we saw.

– Search-based technologies. Most of today’s commercial customer matching systems (Firstlogic, Group 1, Trillium, Innovative Systems, DataMentors) were originally designed to merge records into mailing lists or customer databases. But another set of products has now appeared, with roots in searches against police, medical or immigration files. These include Search Software America, Intelligent Search Technology and ChoiceMaker. Like merge-oriented systems, the search products ultimately compare records to determine whether they match. But the search systems generally apply more sophisticated mechanisms when selecting records to compare, and are less dependent on parsing and standardization for accurate results.

In particular, the search systems are noteworthy for considering the frequency of individual names within their logic. In practical terms, this means they require more precise matches against more common names. Frequency is most often used during the initial selection of records to submit to a detailed matching process. This reduces the number of records that must be matched against common names–an important consideration for online search systems–without excessively limiting the candidates compared to unusual names. Frequency is also sometimes used within the matching logic itself, on the theory that unusual names are more likely to be misspelled, misformatted, or otherwise mangled during data capture and processing. (SAS DataFlux and Ascential QualityStage [originally Vality Integrity] also use frequency in these ways, although they did not start out as search systems.)

To reduce their reliance on parsing and standardization, search systems may match against all words in a name field, rather than attempting to identify family vs. given names, or match against both original and standardized addresses rather than the standardized address only. This typically requires storing several versions of each reference record–another way to improve online response time, although at the cost of higher storage and initial processing expenses.

Search system vendors claim to be more accurate and easier to set up than their merge-oriented competitors. Of course, the competitors disagree. Both sides point to tests that support their position, but the painful truth is that results will vary greatly depending on the data, applications and technicians involved. So careful buyers have little choice other than to run tests of their own.

– Surveillance applications. It’s old news that customer matching systems are now connected to online operational systems to identify customers during order entry, technical support, and customer service transactions. Such applications came naturally to search systems, while merge-oriented systems added real-time candidate selection and matching against reference databases to support them.

These capabilities have in turn enabled new applications to spot fraud, money laundering and terrorists. These are based on existing matching functions but still require product extensions for new user interfaces, audit trails, and reference list maintenance. Some also need new matching methods to find relationships among individuals and organizations. Surveillance is a new business for many matching vendors and often involves buyers who are not their traditional customers. But several firms report it already accounts for a substantial revenue stream.

– Better handling of international data. Both customer files and surveillance systems must increasingly contend with names and addresses of people from outside the United States. These often involve spellings, rules and formats that are quite different from U.S. or Western European standards. Most matching systems can accommodate these by applying specialized rules and data tables to their existing processes. While this is still a major investment, major vendors are increasingly finding it worth making.

– Easier system configuration. Matching has traditionally been part of business processes, such as loading a data warehouse, that are set up and managed by technical users. The matching system itself would often be tuned once, during implementation, by specialists from the vendor or an outside consultancy. As a result, buyers focused on system accuracy and flexibility but paid little attention to ease of deployment or revision.

Today, matching systems are used for more applications and requirements can change quite frequently. Companies also have less time and money for implementation and maintenance. Vendors have responded by adding graphical user interfaces, reports and other features to speed and simplify deployment. In theory, this could allow non-technical users to set up their own processing streams and business rules–although in practice such users rarely take on these tasks by themselves.

– Expansion beyond name and address matching. Matching systems are increasingly being asked to parse, standardize and match non-name and address data such as product IDs. This reflects the continued extension of business intelligence systems and the greater familiarity of technology departments with matching system capabilities. Streamlined configuration processes and matching methods that work despite minimal parsing and standardization are important contributors to this trend. Tight budgets, which make it harder to purchase new tools rather than repurpose existing ones, may also have something to do with it.

– Extended product suites. Several matching vendors have extended their products to provide input profiling, transformations, loading, consolidation, enhancement and analysis. Combined with the mix of batch and real time processing capabilities, this allows them to position the systems as a “complete data quality solution”–with the precise definition of “complete” always conveniently corresponding to the particular set of features the vendor happens to offer.

This type of expansion is typical of the software industry, and in particular of specialized products that become more widely adapted. The fundamental challenge faced by vendors in this situation is that once their tools become well understood, larger vendors of related products can easily add some version of the specialized capability and then squeeze the specialists into an ever-shrinking “best of breed” niche. Matching vendors already face precisely such competition from Ascential and SAS, which have both purchased and integrated specialized matching systems. Potential competitors including Informatica and Siebel have so far chosen to partner with matching vendors rather than acquire or replace them, but there are no guarantees this will continue.

Line extension is therefore essentially a defensive move. The specialists seek a large enough presence to justify remaining part of corporate infrastructures or, failing that, enough revenue from a smaller number of clients that they can still survive. Historically, such expansion has rarely succeeded: small software companies have a hard time supporting a large product line, and big competitors have an overwhelming marketing advantage. So the strong likelihood is continued consolidation of the matching market, as small companies drop out or, more likely, are acquired by larger firms for their technology. In other words, the changes we saw this year are just a hint of the changes yet to come.

* * *

David M. Raab is a Principal at Raab Associates Inc., a consultancy specializing in marketing technology and analytics. He can be reached at draab@raabassociates.com.