David Raab Article Archive

Whether you call it hosted, on-demand, software-as-a-service or cloud computing, the idea of using shared systems managed by someone else is being tried for every conceivable technology function, including business intelligence.    But technologists are famous for pushing good ideas beyond their natural limits.  So it’s worth asking whether on-demand business intelligence truly makes sense.

In some ways, the fit is quite good.  Business intelligence systems are primarily analytical, not operational in the sense of order processing or billing.  This means a business intelligence system need not be tightly integrated with the other enterprise processes and can endure unexpected downtime without catastrophic results.  Integration and availability are both significant issues for on-demand products.  There is also a long tradition, at least among marketers, of hiring outside service bureaus to build and manage their analytical databases.  This is primarily because internal IT departments have had other priorities and lacked the specialized skills needed for such systems.  This history of relying on outsiders makes it easier to consider using another external service.

But on-demand business intelligence faces hurdles as well.  The most daunting is sheer diversity. Popular on-demand applications, such as collaboration, content management and sales automation, are largely the same from company to company.  Not so with business intelligence, where most implementations integrate a unique set of data sources into a custom-tailored structure.  This makes standardization difficult and requires personal attention from technical experts.  Yet savings from standardization and automation are precisely how on-demand systems make their money.  Without them, on-demand has no particular advantage over conventional approaches.

On-demand business intelligence vendors including Autometrics, Birst, BlinkLogic, GoodData, LucidEra, oco, OnDemandIQ, and PivotLink have addressed these issues in different ways.  Strategies include:

- predefined sources.  Vendors can focus on specific applications, such as analyzing data from Salesforce.com.  This lets them prebuild connectors to the source data, map their contents into a standard database design, and create standard reports.  Technically this is a simple approach.  In fact, it is no different from the vertical application packages commonly built for conventional business intelligence systems.  Nevertheless, it lets vendors provide substantial benefits to their clients.

- tools to let users do the work for themselves.  Common examples are wizard-driven interfaces to set up data integration rules and define reporting requirements.  This is a relatively easy solution from the vendor perspective, since it generally boils down to putting a new interface on an existing tool.  However, it places a substantial burden on clients who may lack the time or skill to do the work.  Vendors who take this approach must supplement it with staff experts who help clients when needed.  Although relying on staff experts begins to look uncomfortably similar to a conventional service bureau, the vendors would argue that their tools allow both clients and staff to be significantly more productive than traditional alternatives.

- automated tools for design and integration.  Several vendors have built systems that automatically import client data, analyze it, and create appropriate data models and integration processes.  Although business and technical experts should review the results before implementation, the automated tools do have the potential to provide a reasonable first pass at the system.  This gives the experts something to review and the end users something they can work with right away.  Automated design is the most technically demanding approach to on-demand business intelligence, but it also tackles the twin issues of variety and labor most directly.

- specialized analytical databases.  Columnar and in-memory database engines are more efficient at analytical processing than conventional products like Oracle, DB2, and SQL Server.  This lets business intelligence vendors build successful systems without fine tuning each implementation, and lets the system accommodate new data elements and queries without major redesign.  Specialized databases nicely complement automated design tools by allowing simpler database designs, which are easier to automate; providing adequate performance without optimization; and allowing easier adjustments to the automated system’s recommendations. The analytical databases also run on less expensive hardware than conventional database technology, supporting the fundamental on-demand goal of lower costs.

- automated opportunity discovery.  This addresses the ultimate roadblock to business intelligence adoption: users don’t always know what to do with the results.  Automated data evaluation tools be adapted to mine for significant business information and present the findings to users.   Although conventional software and staff experts can do something similar, automated discovery is particularly value for on-demand systems because it proves the system’s value almost immediately.  In fact, if the vendor has successfully eliminated labor from the rest of the implementation process, it may be able to make automated analytics part of the sales cycle, reducing the buyer’s risk to almost nothing.

Can any of these strategies simplify the delivery of business intelligence systems enough to make them a viable on-demand business?  Bear in mind that the vendors are competing not only with each other, but against the on-premise and on-demand offerings of conventional business intelligence providers; analytical offerings integrated with Customer Relationship Management and Enterprise Resource Management suites; and solution mash-ups that combine on-demand services such as data integration, data storage, and processing power.  All these vendors have the same underlying technologies available.

Although the jury is still out, I believe the answer is yes.  Right now, offerings tied to specific data sources and applications are most likely to succeed because they combine inherent labor savings with the efficiencies of on-demand infrastructure.  Longer term, the greatest potential lies with automated tools and analytical databases.  These should let developers create an integrated process that requires little effort from the vendor or its clients, and is substantially more flexible and cheaper than conventional alternatives.

As with any technology, on-demand business intelligence will be limited at first.  For now, it’s worth testing the automated systems to understand what they really can do.  Even if the current reality doesn’t quite live up to the promises, bear in mind that the systems will grow over time.

*                            *                           *

David M. Raab is a Principal at Raab Associates Inc., a consultancy specializing in marketing technology and analytics.  He can be reached at draab@raabassociates.com.

Last month’s column discussed how to identify the system functions you need in a demand generation system.  But functions are only one piece of the puzzle, and many vendors argue that their real advantage actually lies elsewhere, in superior “ease of use”.  How do you measure that one?

This is a serious challenge.  Functionality is relatively straightforward: either a system has a particular capability or it doesn’t.  But “ease of use” is more subtle.  Formal, objective measurement is difficult and expensive, so most people end up relying on subjective assessments instead.  And even subjective assessments are hard to organize.

To make any sense at all of the topic, we need to take a couple steps back.  First, “ease of use” is one component of a larger and ultimately more important concept of “usability”.  This is defined in different terms but generally includes ease of use, ease of learning, and functionality.  Second, both ease of use and usability are highly contextual: they can only be measured for specific tasks for specific users in specific circumstances.  This makes perfect sense if you think about it for a moment: a system may be quite easy to use for one task, yet very difficult for another.  But it means that a single “usability score” makes much less sense than a single “functionality score” (not that functionality scores make much sense to begin with—the point of last month’s column was to measure only the functions that matter to you).

The good news is that accepting the contextual nature of usability is the first step towards a reasonable way to measure it.  Specifically, it hints that you should start by defining the contexts you need to consider.  Then you can actually assess usability within those contexts.

We’ve already listed three broad types of contexts: tasks, users and circumstances.

- The tasks you need to consider are essentially the same ones identified in the use cases that drive your functional requirements.  Usability analysis does add one new dimension to these: measuring the effort to do a task the first time compared with the effort to do it repetitively.  The difference is set-up effort, which applies to the first iteration only.  This turns out to be a significant differentiator among demand generation systems, since some are optimized for simple, one-off campaigns (not much set-up but little reuse) while others are best at creating many variations within a fixed theme.

- Users, of course, come in many varieties.  Some dimensions to consider are: experience with the system; frequency of system use; skill sets (marketing, analysts, technologists, etc.); and system access (end-users vs. administrators).  You will first identify the characteristics of your users, recognizing that you will probably have several sets of users who fall into different groups.  Then, determine which types of users will perform which tasks, bearing in mind that some tasks may be shared across different groups.    For example, administrators who are technically skilled and frequent users may set up program templates that are then completed by infrequent end-users who are primarily marketers.

- Circumstances vary as well: will your users be focused solely on the demand generation system or will they be in chaotic environment with many distractions?  Will they be under extreme time pressure or be able to plan ahead?  Is there some leeway for error or must everything be perfect the first time out, perhaps for regulatory reasons?  The answers bear on system attributes such as display style, alert functions, error checking capabilities, approval workflows, versioning, and security.  Again, different tasks will likely be performed in different circumstances.

Once you’re worked through these issues, your original task list will now be extended to include information on which types of users will perform each task, and in what kinds of circumstances.  This allows you to assess the usability of each task against the right criteria.

This still hasn’t answered the question of how to do the assessment itself.  In an ideal world, and sometimes in the real world when the stakes are high enough, you would install a test version of the software, train the appropriate users, and let them work with it.  But most evaluation projects don’t have the time or resources to make such an investment.  Yet even if you’re limited to what can be accomplished in a couple of vendor demonstration sessions, you can ensure that you are looking at usability in the right context.  Here are some specific suggestions:

- have the vendor demonstrate tasks that would be performed by expert users.  This assumes that the vendor’s demonstrator, either a salesperson or engineer, is herself likely to be an expert.  The demonstrator will surely make things look easy—but your own experts will eventually learn the same tricks, so that’s okay.

- have the vendor show you how to perform tasks that would performed by casual users.  That is, you should operate the system rather than watch.  If you are already an expert, it might be better to have a typical end-user at the controls instead.  The goal is to see how well someone unfamiliar with the system can work with it, at least for tasks likely to be done by such users.

- look for features relevant to the appropriate user groups.  Expert users can invest the time to learn system shortcuts, set up reusable templates, and make correct choices in unstructured environments.  Casual users rely more the intuitive choice being the right choice, expect more guidance and error-checking, and may prefer graphical interfaces.  Casual users may also need to rely on templates and other components produced by the experts.

- when different kinds of users will share the system, look for options to tailor the interface to individual needs.  These include capabilities to turn help messages off and on, to limit the capabilities offered to casual users, and to present predefined workflows.

Be sure to prepare a specific list of these items in advance of the demonstration itself, so you know exactly what to look for and have a place to record your observations.  The more structured your process, the more you’ll be able to cover and the more clear you’ll be on what you actually saw.

Finally, remember to look beyond the demonstration itself.  Reference clients are a particularly important source of insight.  Anybody the vendor recommends is almost certain to be happy (although slip-ups are more common than you might think), but you still need to assess whether the reference context (tasks, users and circumstances) is similar to your context.  If the vendor can’t connect you with someone similar, you’ll have to work harder to assess the product directly.  You can also look at other elements of the customer experience, such as training, support and user forums.  These have the advantage of being objectively measurable, but bear in mind that they often have more to do with the size and maturity of the vendor than the quality of the product itself.

*                            *                           *

David M. Raab is a Principal at Raab Associates Inc., a consultancy specializing in marketing technology and analytics.  He can be reached at draab@raabassociates.com.