Business Intelligence/Introduction

“	The availability of documents in machine-readable form is a basic requirement of the system. Typewriters with paper-tape punching attachments are already used extensively in information processing and communication operations. Their use as standard equipment in the future would provide machine-readable records of new information. The transcription of old records would pose a problem, since in most cases it would be uneconomical to perform this job by hand. The mechanization of this operation will therefore have to wait until print-reading devices have been perfected.	”
—H. P. Luhn

Business Intelligence and Business Intelligence System

Rapid advances in computer technology allow business intelligence (BI) systems to provide managers with access to a tremendous amount of data. To function these systems combine complex front-end software with ETL capabilities that extract enormous amounts of data. At the heart of these systems are huge enterprise data warehouses that can populate a possible infinite combination of advanced reports, OLAP cubes and datasets for data mining. The underlying belief is that technically advanced systems are the most important drivers of effective decision making. Based on this belief BI vendors focus on technologically advanced systems while paying relatively little attention to whether these systems meet the needs of decision makers.

The drive towards technological sophistication and away from improving decision making is one of the reasons for the low success rates (50%), high costs and time overruns associated with BI projects. While vendors and information technology departments get the technology right they often fail to deliver a product that is useful to management. Executives and managers will reject BI systems that do not facilitate efforts to evaluate and improve strategic and operational effectiveness. What practitioners fail to understand is that technologically advanced software, by itself, does not allow management to perform these vital functions.

Effective business intelligence systems must account for the goal oriented behavior of decision makers. One way to incorporate the purposive behavior of decision makers in BI systems is the use of the frame as the basic building block of a pervasive BI system architecture. It does this since a frame is a "scheme of interpretation" in which the particulars of events and activities are organized and made sensible (Goffman 1974). The frame organizes the world according to the goal oriented behavior of the frame owner. Specifically, the frame captures the two parts of decision makers' goal oriented behavior. The first are the objectives disseminated from the ESM in the form of scorecards. These provide the goals that guide the activities of the business unit. The other part is the experience of the business unit leader. It is their experience that provides the guidance for how to achieve the objectives. The goal oriented behavior of the frame owner is codified in a business unit strategy map based on the objectives that the unit must achieve along with the means of achieving those objectives.

At the heart of this system are strategy maps and scorecards. We propose strategy maps can be used to organize the company as a system since each scorecard contains the necessary elements to create a control system. A control system requires a measuring unit, norm, comparator and correction unit. The analogs in a scorecard are the measures, budgets, reports and initiatives. The importance of the control system is that it allows the business unit leader to evaluate and correct issues that impede operational effectiveness. Once management is certain they are operating effectively they can report to executives.

An important aspect of this system is that each BI subsystem can operate independently of all other subsystem. As a system each BI subsystem is both unique and complete. For this reason the creation of a pervasive BI system begins with the creation of BI subsystems at all levels of the organization. Each one can function as a distinct unit prior to the creation of the system. Further, it is possible to fix problems in any one subsystem without interfering with the operations of the whole system.

Upon completion of the subsystems these frames are then combined into a framework. A framework stacks frames horizontally and vertically. Executives can use the framework to determine if all business units are operationally effective. Each executive can have a BI system that allows him/her to access information from any subsystem. This is because they have a system built based on their frame. They use this subsystem to control other subsystems.

This pervasive business intelligence system provides two important contributions. First, it significantly increases the likelihood that they system will not only be deployed but will be accepted by users. The system is design for decision makers based on their needs as outlined in their business unit strategy map. They outline the reports that are necessary based on the combination of objectives from their superiors' strategy map and their experiences.

The other important contribution is the ability to evaluate both strategic and operational effectiveness. As Porter (1996) argues operational and strategic effectiveness are both necessary for success. However, it is first necessary to achieve operational effectiveness. Each subsystem facilitates the analysis and reporting of business unit operational effectiveness. After ensuring operational effectiveness executives can then examine strategy effectiveness.

Finally, it is easy to make changes to the system if it is not effective. For instance, if a business unit is not achieving operational effectiveness then executives can determine why it is not successful. Remedies can be taken, such as allocating further resources or attempting to align frames in the case that operations does not share the same beliefs as management. If the organization is operationally effective but strategically ineffective then executives need to reformulate the strategy. Once they make the necessary strategic changes they then disseminate a new strategy map and cascade scorecards. This allows the system to change.

Most definitions agree that BI "refers to skills, technologies, applications and practices used to help a business acquire a better understanding of its commercial context. Business intelligence may also refer to the collected information itself." [1]

In addition, "Common functions of business intelligence technologies are reporting, OLAP, analytics, data mining, business performance management, benchmarks, text mining, and predictive analytics." [2] Finally, "Business intelligence often aims to support better business decision-making. Thus a BI system can be called a decision support system (DSS)." [[3]

These definitions focus on the skills and functions. However, prior to engaging in the activities involved in BI or using the technologies there must be a system in place. There is a difference between BI activities and BI systems. We take the comprehensive view that building and maintaining a BI system are closely tied with using the system.

Business Intelligence

Business Intelligence allows the decision maker to understand their business and the business environment in order to make informed decisions. Decision making requires evaluating performance (what happened), testing hypotheses (why and how things happened) and predicting future events (what may happen). Stated simply management needs to know if their strategies are sound and if they are being carried out. Most formally a Business Intelligence system allows the user to answer any of the following questions:

1. Why did it happen?: X-->Y
   • Did X cause Y to happen?
   • If we do X then will Y happen?
2. How did it happen?: X₁ -->Y₂
   • How did X produce Y?
   • Can we be certain that X will produce Y, or that Z is actually producing Y?
3. What happened?: Y₁ versus Y_y
   • What happened versus what we expected to happen
   • Assuming X produces Y, are we really doing X?
4. What may happen?: X_f -->Y_f
   • If X_f occurs in the future then will Y_f also occur?
   • Assuming we did X and it produced Y, can we assume that continuing to do X will continue to produce Y?

Definitions:

X: Independent variable

Y: Dependent variable

X₁: Specific instance of X

Y₂: Specific instance of Y

X_f: X in the future

Y_f: Y in the future

The purpose of this wikibook is to present a process for developing a Business Intelligence (BI) system that will allow the analyst to ask and answer these questions. The process of creating this system is outlined in sections 1 and 2 of this book. This book takes the reader through each stage (planning, development and production) of the process, step-by-step, in detail. Before moving forward it is necessary to define some concepts. It is important to understand exactly what is meant by a system. One way to look at a system is that it is composed of both an architecture (blueprint) and infrastructure (people, process and technology). Architecture is "the set of rules or structures providing the framework for the overall design of a system or product." (Poe et al. 1998) Technical Infrastructure is defined (Poe et al. 1998) as the "technologies, platforms, databases, gateways, and other components necessary to make the architecture functional within the corporation." Other components are assumed to mean the people and processes necessary to deploy and maintain the infrastructure. This book therefore defines infrastructure as the people, process and technologies necessary to implement a BI architecture. This book adopts the current methodology of building a Business Intelligence system from the top-down. Top-down refers to understanding the business strategy and then building the system so the decision makers can determine if the strategy is effective and that the strategy is being successfully implemented. The framework approach explicitly embeds strategy in the Business Intelligence system architecture. This is accomplished by first formalizing the strategy in the form of diagrams and using these diagrams to create the system. In this way the diagrams become part of the Business Intelligence architecture. The explicit statement of strategy is crucial because the purpose of Business Intelligence is to test formal theories and hypotheses about strategy in order to validate the strategy or to monitor performance. As each part of the strategy diagram is verified the company can have confidence that aligning the day-to-day activities with the strategy will help the company create value. Once this is accomplished the system allows management to monitor business performance. Further, if the company is not profitable and the strategy is indeed being executed effectively then management can take steps to determine reformulate the strategy.

Definition of BI

The purpose of Business Intelligence (BI) is to provide decision makers with the information necessary to make informed decisions. The information is delivered by the Business Intelligence system via reports. This book focuses on the architecture and infrastructure needed to deliver the information. An architecture is a set of rules or structures providing a framework for the overall design of a system or product (Poe et al. 1998). The BI system includes the following parts:

• Interested parties and their respective information needs
• Input of data
• Storage of data
• Analysis of data
• Automatic and selective dissemination of information

A BI system includes the rules (architecture) providing a framework for the organization of the technologies, platforms, databases, gateways, people and processes. To implement an architecture the Business Intelligence architect must implement an infrastructure. Technical infrastructures are the technologies, platforms, databases, gateways, people and processes necessary to make the architecture functional within the corporation (Poe et al. 1998).

In sum, decision makers need reports that deliver the information that allows them to understand his/her organization and the world in order to make better decisions.

Given these functions of the BI system the most enduring definition focuses on the system, tools, technology, process, and techniques that compose these four elements that help decision makers understand their world. BI augments the ability of decision makers to turn data into information by aiding in extracting data from data sources, organizing the data based on established business knowledge and then presenting the information in a manner that is organized in a way to be useful to the decision maker. It merges technology with knowledge in order to provide useful information to management as quickly as possible.

In sum, Business Intelligence system includes the rules (architecture) that outline how to organize the parts of the system (infrastructure) to deliver the information needed to thrive in a competitive market (business) or to provide the best service to the people (government) (Poe et al.1998). Regardless of the technology, which are simply tools, the core of a Business Intelligence system has and will not change.

History of BI

In the beginning...

Leonardo da Vinci, the famous Italian know-it-all, toyed with the idea of an open source BI platform, but abandoned the project when he discovered that open source and Business Intelligence needed to be invented first.

Pentaho website, largest open source Business Intelligence provider

In reality, H. P. Luhn (1958), in writing on Business Intelligence systems, wrote “The system described here employs rather advanced design techniques and the question arises as to how far away such systems may be from realization.” The system needed techniques and technologies much more advanced than were available at the time. He noted:

“	The availability of documents in machine-readable form is a basic requirement of the system. Typewriters with paper-tape punching attachments are already used extensively in information processing and communication operations. Their use as standard equipment in the future would provide machine-readable records of new information. The transcription of old records would pose a problem, since in most cases it would be uneconomical to perform this job by hand. The mechanization of this operation will therefore have to wait until print-reading devices have been perfected.	”
—H. P. Luhn

In 1989 Howard Dresner proposed BI as an umbrella term to describe "concepts and methods to improve business decision making by using fact-based support systems." It was not until the late 1990s that this usage was widespread (Wikipedia).

The most rapid (but not important) developments in the history of BI focus on vendor innovation. Architecture changes very slowly and existed before the technology was available to create BI systems. The creation of the technology, and the subsequent rapid development of tools by BI vendors, requires

Vendors

Business Intelligence infrastructures rely on technology to turn data into intelligence. Vendor innovations are the heart of innovation. The rate of innovation has continued to rise since the 1990's. Not only do vendors create new products and improve on existing ones but immitation means technology diffuses quite quickly between companies.

At the same time vendor products are becoming more advanced ease-of-use is also increasing. These seemingly counterveilling forces are actually complementary. This is because there is general agreement the needs of the user has not changed substantially. For instance, as Poe et al. (1998) note "While the products and technologies surrounding and complementing data warehouse implementation have become much more refined, the process of actually building a data warehouse has not significantly changed." The same can be said of most Business Intelligence system components. User needs and architecture changes much slower than technology but technology must be tailored to the needs of the user. Therefore, the focus should always be on the enduring parts of the BI system without focusing on any one vendor or technology.

Vendor Products and Tools

Business intelligence begin to evolve in 1990's. Business intelligence is now a mature field. Mature in this context means that there are a core set of products and services offered by vendors.

“	Business intelligence tools are a type of application software designed to report, analyze and present data. The tools generally read data that have been previously stored, often, though not necessarily, in a data warehouse or data mart.	”
—Wikipedia (Business intelligence tools)

The key general categories of business intelligence tools are (Wikipedia Business intelligence tools):

• Spreadsheets
• Reporting and querying software - are tools that extract, sort, summarize, and present selected data
• OLAP
• Digital Dashboards
• Data mining
• Process mining
• Business performance management

Each tool can be sold in a variety of ways, include as standalone tools, within a suite of tools, as components of software targeted to a specific industry, within a web-based Business Intelligence 2.0 infrastructure, or packaged into data warehouse appliances.

The most important development in the vendor space is acquisitions. The recent trend in the marketplace is for larger companies to acquire smaller companies (Richardson et al. 2008). This has consolidated the BI market share. The largest include:

• 2008 - IBM purchases Cognos for $5 billion
• 2008 - SAP purchases Business Objects for € 4.8 billion
• 2007 - Oracle purchases Hyperion for $3.3 billion

Note that a few business intelligence vendors own a substantial share of the market. The following figure shows market share by vendor between 2006 and 2007.

BI Market Share by Vendor(revenue in millions of U.S. dollars)

Figure: BI Market Share 2007

Purpose of businesses

Finally, this text makes a simple assumption regarding the purpose of for-profit businesses. The purpose of businesses is to generate profits. The best companies achieve superior profitability over time. They do this through strategic positioning, meaning performing different activities from rivals' or performing similar activities in a different way. Specifically, to achieve profits companies can focus two different types of activities. The first is for companies to provide goods or services with greater value, charge higher prices, thereby increasing revenue. The second type of activity aims to provide goods or services at a lower cost, reduce costs, thereby increasing profitability (Porter 1997).

This approach is also useful for existing systems. The goal of this book is to present a way to think about BI systems. Companies and their business environment constantly change. Competitive businesses must change in order to remain viable. BI systems allow companies to determine if the environment is changing and if their strategy will remain effective in the new landscape. BI should be considered a constantly evolving system and the process presented in this book should be applied iteratively in order to continuously update the BI system.

Future of BI

The Gartner Group makes a couple predictions about the future of BI. First, between 2009 and 2012, more than 35 per cent of the top 5,000 global companies will regularly fail to make insightful decisions about significant changes in their business and markets (Gartner Group). Faced with the need to make better decisions companies will seek to increase their investments in BI systems.

Another important prediction is that business units will control at least 40 percent of the total budget for BI despite the fact that IT departments are responsible for building and maintaining the BI infrastructure (Gartner Group). The reason for this situation is that business users have lost confidence in the ability of IT departments to deliver the information they need to make decisions (Gartner Group). Faced with these problems it is no wonder that the business unit will take it into their own hands to create the BI infrastructure. A BI system that does not meet the users' needs will fail. This is because the failure of a BI system to deliver useful information can have a strong, negative impact on decision making.

However, IT departments are charged with building and maintaining IT infrastructures they are better suited to this role. Again, although IT organizations excel at building BI infrastructures, business users have lost confidence in their ability to deliver the information they need to make decisions.

Not surprisingly there are risks when business units create their own BI infrastructures.

“	By making purchases independently of the IT organization, business units risk creating silos of applications and information, which will limit cross-function analysis, add complexity, and delay to corporate planning and execution of changes...IT organizations can overcome this by encouraging business units to use existing assets and create standards for purchasing classes of packaged analytic applications that minimize the impact of isolated functions.	”
—Garner Group

Building on existing resources is an excellent way to develop a BI system. However, this does not solve the problem. An important step is that business units and IT departments must accept that BI systems are composed of architecture and an infrastructure. The business unit should lead the creation of the architecture whereas the IT department should lead the creation of the infrastructure. Ideally, the company will have a cross-function BI unit that creates, maintains and updates all BI systems. This allows each group to perform the function they are best at and use resources as efficiently as possible.

This book presents a methodology to build this type of system. By dividing the building of frames and frameworks into three distinct stages it allows the business unit and IT department to focus on what they do best. Further, this approach encourages and demonstrates a process by which business units and IT departments can work together to build a BI system.