Business Intelligence is Dead – Long Live the Highly Evolved Business, Part 3

Originally published May 2, 2007

In Part 1 of this series, I introduced the concept of the highly evolved business and said that a service-oriented architecture (SOA) will be key to our ability to achieve that goal. Part 2 provided an overview of the concepts, business drivers and technical implications of SOA. In this third article of the series, I’ll share my best guesses on how SOA and business intelligence may coexist – or otherwise – in the future. 

As previously described, a highly evolved business is one where the operational, informational and collaborative environments as well as the people, process and information layers are seamlessly interconnected and working together to provide a single, integrated and flexible IT infrastructure to support on demand business. This is illustrated in Figure 1. 

Figure 1

SOA was defined as an approach to decomposing business workflows (or applications in the broadest sense of the word) into discrete, well-bounded services and providing the ability to recompose them in a flexible but well-integrated manner. These characteristics enable, and indeed are prerequisites, for adaptability, responsiveness and innovation in today’s fast-moving business world. 

How Can SOA Benefit Business Intelligence?
In the early days of data warehousing, business users were so happy to receive any real, consolidated view of the way the business was operating that they were willing to see information that was days or more out of date. This was good news for IT, because reconciling data across multiple operational systems was slow and laborious, and many operational applications did not have internally reconciled data available until various “end-of-day” processes had run – often many hours after the daily business close. As a result, data extraction, transformation and loading (ETL) essentially became a batch process. 

Of course, over the years, users’ expectations for timeliness grew and a modern on-demand business simply cannot afford the luxury of waiting days or even hours for such a consolidated view. For many years now, vendors have used terms such as “active,” “dynamic” or “real-time” warehousing to indicate how the warehouse tools could meet these needs. However, in truth, no matter how close to real-time the ETL process became, the internal reconciliation processes of the operational systems remained a bottleneck. 

Enter SOA, which enables the retooling of operational applications and allows them to become more “real-time” themselves. Of course, such behaviour within and between operational systems is also absolutely required for an on demand business in any case. But, the application of SOA within the operational environment also enables business intelligence to provide more timely information than previously possible. On the other hand, it also becomes increasingly difficult to draw a distinction (especially in the minds of end users) between operational reporting and business intelligence. 

At a deeper technical level, SOA is likely to drive a fundamental change in the way ETL works. Today’s ETL is largely about extracting data from databases after it has been processed and stored by the owning applications. However, as SOA develops, the message queue can become a much more important source of data for the warehouse. In an SOA environment, all the services communicate via messages on an enterprise service bus. This bus provides a variety of services itself to assure message delivery, integrity, security, and so on. Clearly, such services are important for ETL too, and we can see that ETL tools will begin to use them as they become available. In addition, we may pose the question: Why should the ETL tool wait until the data is stored in a database before going to extract it? Since the enterprise service bus messages contain the data the warehouse ultimately needs, we can envisage that ETL processes could eventually start from the queue. 

Ultimately, ETL could move from being a batch process to a set of infrastructure services on the enterprise service bus. These services would revolve around data cleansing, validation and even reconciliation, where some messages would have to be stored temporarily until the corresponding messages arrived from other services. Many of these services will be shared with services in the operational environment, so the business intelligence development team will no longer be the sole owner of this area of tooling. 

The application of SOA principles in the operational world also offers great hope in one of the most intractable problem areas of business intelligence for many years – metadata. Business intelligence has always struggled to create and maintain sufficient and accurate metadata both for the internal control needs of the warehouse as well to enhance end user understanding of the information delivered. While some of the blame lies with warehouse projects themselves, which often reprioritised metadata to the end of the list when development came under stress, a much larger share of the problem resided with the source operational systems that had little or no need for active or even explicit metadata themselves. Because the data warehouse is dependent on the operational systems for a significant portion of its metadata needs, business intelligence systems always suffered from a dearth of good metadata. 

Again, the application of SOA in the operational environment comes to the rescue of business intelligence. Defining business services and flexibly linking them cannot be done without explicit metadata describing the data meanings and structures. Automated workflows between loosely coupled services are fully dependent on accurate and active metadata. So, once again, business intelligence systems can benefit as a side-effect of these enhancements to the operational systems. 

However, as in the case of ETL, there is a small sting in the tail for business intelligence. Traditionally, enterprise data modelling – the source of much metadata – has largely been the preserve of the data warehouse development teams because it was really the data warehouse (and the operational data store) that needed such overarching models. Implementing SOA fully in the operational environment will require a significant level of enterprise data modelling to ensure that all the services “speak the same language”: that is to say that they all have a shared understanding of the data elements they transfer between one another. There is no doubt that the data warehouse will benefit from this, but the business intelligence development team will lose yet another area of specialization. 

SOA is thus likely to provide significant benefits to business intelligence by fixing some of the longstanding problems of getting and understanding data from the operational environment. However, how these benefits will be delivered has the potential to be quite disruptive to existing business intelligence perceptions and tooling. As SOA changes the way operational systems work, it will also break down the boundaries that end users have traditionally seen between operational and informational activities – as SOA enables the highly evolved business, it will dissolve the characteristics that made business intelligence different. Furthermore, from an IT viewpoint, SOA has the potential to disrupt some of the key technologies that today are specific to business intelligence, such as ETL, metadata and enterprise data modelling. 

Can Business Intelligence Benefit SOA?
So, I’ve argued in part 2 of this series that from an end user perspective, SOA will drive business intelligence, operational and office computing together into a single user environment where business intelligence (BI) is no longer seen as the preserve of the business analysts. In the previous section, I’ve posited that SOA is about to disrupt most of the key underlying technologies that distinguish informational for operational systems development. What, then, is the future for business intelligence developers, managers and consultants once SOA takes a firm hold? Are they to be consigned to the scrapheap of the IT industry? 

I believe the answer is NO! 

There is an important set of skills that data warehouse developers have long had to cultivate that will not only be needed, but which will be vital in the new SOA world. And, in many cases, the current inhabitants of planet SOA don’t even know they lack nor need these skills. I’m talking about the ability to do cross-enterprise IT development. 

Building a data warehouse has always required a subtle blend of deep technical expertise and political nous. Data warehouse development is technically challenging because of the mix of new development and systems “archaeology” needed. The quirks of older systems developed under different technological constraints have to be understood and accommodated. New tools and techniques need to be assimilated. And conflicting technical requirements must be balanced. But perhaps more importantly, good data warehouse developers learn early how to balance changing and often conflicting organisational needs. Trade-offs have to be made in what to deliver when, to which part of the organisation, and in what balance between infrastructure development that has long-term advantages and meeting immediate business needs with temporary solutions. 

These skills arise because data warehouse development is an enterprise-wide, cross-functional, long-term project that must deliver both infrastructure and business value in parallel. Data warehouse projects also have significant dependencies on existing applications and infrastructure as well as on ongoing projects that are outside the control of the development team. 

Delivering a service-oriented architecture is exactly the same. So, while business intelligence as we currently know it may disappear, the expertise and knowledge gained through data warehouse projects will continue to stand you in good stead – not only in developing the highly evolved business, but also in the broader arena of SOA implementation. 

SOURCE: Business Intelligence is Dead – Long Live the Highly Evolved Business, Part 3

  • Barry DevlinBarry Devlin
    Dr. Barry Devlin is among the foremost authorities in the world on business insight and data warehousing. He was responsible for the definition of IBM's data warehouse architecture in the mid '80s and authored the first paper on the topic in the IBM Systems Journal in 1988. He is a widely respected consultant and lecturer on this and related topics, and author of the comprehensive book Data Warehouse: From Architecture to Implementation.

    Barry's interest today covers the wider field of a fully integrated business, covering informational, operational and collaborative environments and, in particular, how to present the end user with an holistic experience of the business through IT. These aims, and a growing conviction that the original data warehouse architecture struggles to meet modern business needs for near real-time business intelligence (BI) and support for big data, drove Barry’s latest book, Business unIntelligence: Insight and Innovation Beyond Analytics, now available in print and eBook editions.

    Barry has worked in the IT industry for more than 30 years, mainly as a Distinguished Engineer for IBM in Dublin, Ireland. He is now founder and principal of 9sight Consulting, specializing in the human, organizational and IT implications and design of deep business insight solutions.

    Editor's Note: Find more articles and resources in Barry's BeyeNETWORK Expert Channel and blog. Be sure to visit today!



Want to post a comment? Login or become a member today!

Be the first to comment!