Data Vault: The Preferred “Flavor” for DW Architecture in BI – Part I
September 21, 2011
Business Intelligence (BI) is todays ‘MANTRA’ chanted by almost every business. Companies want to outsmart the competition. Companies are ready to invest big bucks and human power to build a sophisticated BI system so that they can have the knowledge that others don’t and seize on the opportunities in the market before others do. BI shows the Future Value of Your Business.
BI systems need DATA and every business has terabytes of real data which can provide them with the information and knowledge they need to make the right decisions on time. But the key is to turn that data into information in a timely, efficient and effective manner once the WHAT AND WHY questions are answered i.e., what information is needed, what matters and why that is required. In today’s market, every business is in a RACE. The race to conquer others. The race to generate more gains/profits. The race to foresee the risks early on so that they can be avoided. So time is of the essence here.
An optimized BI system integrates large volume of external and internal near real time data to allow management to create opportunities by making intelligent decisions after performing predictive analysis of their approach on the business. A good BI System is like a GPS. An effective GPS is one that not only shows you a route to your destination but also guides you when you hit roadblock, gives up-to-date external conditions (constructions / traffic) information, provides multiple routes to choose from, suggests you with alternatives for shorter and fastest routes, predict the total time based on your driving behavior, tells you what to expect next etc. Just knowing the path to your destination is not sufficient. You need to know many other factors during the whole ride to reach destination on time and without any hurdles.
For a good integrated BI system, a good Data warehouse architecture needs to be in place. Data warehouse architecture is “an integrated set of products that enable the extraction and transformation of operational data to be loaded into a database for end-user analysis and reporting”. Below are the pictorial representations of different “flavors” of DW architectures.
Methodologies used by different architecture:
Kimball’s DW Architecture – Is based on ‘Bottom-UP’ methodology.
Inmon’s DW Architecture – Is based on ‘Top-Down’ methodology.
Dan Lindstedt’s Data Vault DW Architecture – Is based on ‘HYBRID DESIGN’
The first two design methods have some limitations for Data Warehouse layer such as inflexibility and unresponsiveness to the changing departmental needs during the implementation phase, insufficient auditability of data back to its source system, inability to integrate unstructured data, inability to rapidly respond to changes (organizational changes, new ERP implementations) or difficult to load type 2 dimensions in real time. This is where DATA VAULT came in to rescue. Data Vault follows a ‘HYBRID DESIGN’ methodology which follows ‘TOP-DOWN ARCHITECTURE WITH A BOTTOM-UP DESIGN’.
The model is a mix of normalized modeling components with type 2 dimensional properties. In this model, the DW serves as a backend system that houses historical data which is integrated by the business keys. All data ‘good, bad, incomplete’ gets loaded into the data vault and all the cleansing and application of business rules takes place downstream i.e., out of DW. This means that Data Vault model is geared to be strictly a data warehouse layer, not as a data delivery layer which still requires physical or Virtual star schemas or cubes for Business Users or BI tools to access.
Bill Inmon in 2008 stated that the “Data Vault is the optimal approach for modeling the EDW in the DW2.0 framework.”
In Part 2 and 3, I am going to explain different components of Data Vault and it’s power with the help of some examples. That will clearly explains why the Data Vault should be a preferred “flavor” for different businesses.
- Jyothi Kaparthi
Plug into the Power of the Data Vault
April 1, 2011
In this day and age, it seems to be trendy to gravitate to the flash and splash of the latest and greatest user facing tools to address Business Intelligence issues. Some believe that if they just get a dashboard and a few nifty graphs, all of a sudden they will have “answers” flowing through their systems and into the reports. …Almost like it was magic.
The true Business Intelligence practitioners know better. Most modern systems still suffer under the same age old issues because they are still doing things the same old way. Some of the issues that are still prevalent are ability to change over time as the business changes and the integration of the information problem.
So when you build architecture on the quick and easy solution in a “silo”, you will eventually hit the wall when it comes to adaptability and scalability. So where does one turn when there is a need for speed as well as the ability to support mission critical reporting and analysis needs that must be able to pass audits?
There is a methodology that attempts to bridge the gaps between the typical issues in the current Business Intelligence offerings. The inventor of the data vault is Dan Linstedt (www.danlinstedt.com) where the concepts and rules are specified for successful engagements.
The data vault is not a product. It is not a magic pill that makes all your IT ills go away. It is a comprehensive approach to addressing real world issues with existing implementations. It brings real flexibility and adaptability to the implementation and brings reliability and dependability to the business. And with a team that understands the power of the data vault, you are now able to take your Business Intelligence environment upon which the tools that do the flash and splash can be sourced from.
According to Dan Linstedt, the inventor of the data vault methodology, the challenges around data integration include some or all of the following:
- Definition, or understanding of the data
- Functions or transformations applied to translate the data
- Interpretation of the data
- “MASTER” determination of the data
- Best storage and architecture of the data
- Visualization of the data
- Accountability and auditability of the data
- Traceability of the data
- Overloading (multi-use of single columns and record types) of data
- Historical data with lost definitions
- Data too big
- No change data capture/no audit trail
- Bad indexes
- No control over source feeds, source timing
- Multi-system valuation dependencies
- Missing data
- Changed Passwords
- Mis-aligned access rights
- Overflowing data
- Out-of-range data
- Bad domain data (a date field contains a string…)
Unless there is a comprehensive plan in place to deal with data integration, then it will only be a matter of time before your implementation will begin to suffer under the weight of the problem. Short sighted solutions only mask this issue for a short time, where our customers need a quick solution that will also stand the test of time and change.
And because of the nature of a data vault, this can be done in rapid releases that bring value within a few weeks of embarking on the project. Because of the style, there are now tools on the market that can generate the table and transformation logic. Once you reach this level, then change is transparent and accessible to your user community as well as the IT staff can finally keep close to the change as it is happening in the business.
According to Dan Linstedt, one would expect the following results from pursuing an implementation that included a data vault:
- Manage and enforce compliance to Sarbanes-Oxley, HIPPA, and BASIL II in your enterprise data warehouse (EDW)
- Spot business problems that were never visible previously
- Rapidly reduce business cycle time for implementing changes
- Merge new business units into the organization rapidly
- Rapid ROI and delivery of information to new star schemas
- Consolidate disparate data stores (i.e., master data management)
- Implement and deploy SOA, fast
- Scale to hundreds of terabytes or petabytes
- SEI CMM Level 5 compliant (repeatable, consistent, redundant architecture)
- Trace all data back to the source systems
With the data vault at the core of your Business Intelligence implementation, you are enabling your enterprise to be as nimble as possible without ignoring the core critical issues around data integration and change over time. Your user community will have the chance to grow at the pace dictated by business opportunity unconstrained by the “normal” issues around the traditional approaches.
Over the next few months, I will be going deeper into the components of a data vault, where it fits into enterprise architecture, and the ways to take advantage of the “Power of the Data Vault”. Stay tuned…
Using Business Intelligence to Drive your own Recovery.
June 29, 2010
eWeek published a video describing the value of using Business Intelligence to find and exploit market and revenue opportunities. Great point, and very well worth the 6:49 it takes to view it. Many organizations are using BI to understand some of the basic historical results of their business. It’s the next level of organization who begins to answer questions like the below using their BI toolset:
- What are my customer’s buying is a basic question, but moreover, what products do they buy together?
- Which products do they buy when times are tough?
- What did they buy during the last recovery?
- What aren’t they buying, and what should I recommend they buy?
All great questions, and clearly a value add of a strong BI platform.
eWeek – Using-Business-Intelligence-to-Find-Your-Economic-Recovery
Alignment, Iteration and Business Intelligence
March 25, 2010
For most of the last two decades, LÛCRUM has participated in creating over 100 solutions for some of the most prominent organizations in business and education. In 1998, LÛCRUM published its first full Business Intelligence Methodology, iStream. The word “stream” was used to symbolize the continuous aspect of the software development lifecycle versus traditional “waterfall” SDLC’s. This post is intended to conceptually explain how LÛCRUM’s iStream is a differentiated and unique approach to the development of successful Enterprise Business Intelligence Solutions. After years of focus on the delivery of Data oriented projects, LÛCRUM has continued to refine its methodology, leveraging the continuous learning from each new engagement to benefit the next, and to enrich the iStream process itself.
The first and probably most important non-technical differentiated aspect of iStream is the concept of Alignment. Many consulting organizations and internal IT organizations have some type of design or planning step often called “Envisioning” as an initial step in their development process. This is for good reason: understanding the customer’s end goal or picture of success is critical to the success of the project. At the same time, this does not procedurally support the fact that many individuals are involved in determining the success of a project, and further, in most cases these individuals are not in detailed agreement in regards to what that success looks like, or how it is defined. Alignment takes this into account, and is a prescribed process to ensure a common understanding of the success criteria by the key stakeholders involved in any enterprise project, including department heads and/or the Information Technology department. This includes a focus on ensuring that a miscommunication cannot occur where language is not specific enough, for example in clarifying the accepted definition of the term “Sales” in a company. To explore this a bit, is “Sales” the number of transactions? The dollar volume closed? Over what timeframe? By what channel? (sales people, resellers, distributors, telesales, etc.) As simple as this concept may sound – misunderstandings or assumptions in areas as simple as this are generally a key reason for project failure. In this area, LÛCRUM is unique and differentiated in its development approach.
Another key differentiation of LÛCRUM’s approach, particularly as it relates to Business Intelligence, is in the concept of the iteration of a project. The iStream methodology allows for iteration in the development of the end result, particularly through the recognition that many pieces may make up the whole. For example, related to Business Intelligence; we may begin by working with an individual decision maker, say the VP of Marketing. In working with this person we may offer to them the YourView Instant Analytics solution, allowing them to rapidly see their information in a new way through the combination of several different reports or sources into a single view. Per the YourView solution, this can take place in a matter of days; however by definition it follows the iStream process – however abbreviated – as it is focused on only a single user. When that VP is prepared to create a complete solution for the Marketing department, the initial work now functions as a pilot/proof of concept rolling into the Alignment, Discover and Architect components of iStream for the larger YourView 360 (Data Mart) project. In this fashion, we are “iterating” our development of the data mart through one or more “Instant Analytics” projects. Both projects follow iStream; however the smaller engagements feed into the larger. When that organization is prepared to roll out an Enterprise Data Warehouse – the same holds true, the work that had been completed at the Data Mart level for the Marketing department will now be employed in the Alignment, Discover and Architect phases of the Enterprise Data Warehouse project. In this fashion the work that we accomplish at any level of the Business Intelligence Solution chain is applicable for the next, and all would be accomplished using iStream.
While the items above are not descriptive of the entirety of iStream, nor of the entire list of benefits of the LÛCRUM approach, they are absolutely two of the components of iStream which differentiate it from the plethora of SDLC approaches available in the market, and another aspect of what makes LÛCRUM a unique Business Intelligence Consultancy.
Good enough?
March 16, 2010
When is good enough, well, good enough? I suppose that depends, one old argument says that close only works in horseshoes and hand grenades. Can it work with decision making? How about decision support systems? Is good enough the manually created spreadsheets that over 90% of organizations use for decision support? I would argue that while it’s not good enough, most business decision makers work that way.
To get at the data that most executives feel they need to make accurate decisions, many turn to the manual modification of existing reports, or the creation of their own “Pet” spreadsheet they use almost daily, or certainly many times a week.
In an update to a report cited last spring on this site, a September, 2009 Dartmouth University study suggests that the error rates in formulas on spreadsheets in their study were only .087% of all formulas they audited. HOWEVER, these were in cases where the formula produced the WRONG RESULT, and actually resulted in 87% OF THE SPREADSHEETS REVIEWED having errors in which the spreadsheet then produced the wrong result.
How good is good enough? What if you could reproduce the “Pet” spreadsheet in a true Business Intelligence solution which would ensure that the data and results in the sheet were as solid as the data in your transactional systems in the first place? How much does the wrong data or the wrong decision cost you, or your company? I would argue that “good enough” might just be good enough, if you could ensure that the data was accurate, and mitigated the possibility of error, while increasing the timeliness of the information to the decision maker. We have deployed such systems in a couple weeks’ time leveraging tools like SharePoint, Excel, and other software products that our customers already owned, and quickly delivered a system to our customer where we dramatically increased the accuracy of their information. These solutions form the basis of our iterative approach to Business Intelligence.
Predictive Analytics & Healthcare
February 12, 2010
As a follow up regarding my post yesterday on Predictive Analytics, I wanted to bring attention to an article that Wired Magazine had last November on a predictive concept for “Modeling Human Drug Trials – Without the Human.” Using similar concepts, as well as rules which were indeed put in place by PHD’s, these folks replicated human trials which had taken 7 years of study – in about an hour. Yup, hit run on the computer, and an hour later the results popped up – which according to the article hit 2 of the 4 markers studied perfectly, the 3rd within an approved margin of error, and the 4th was just below the accepted margin of error. Ok, the computer model took 2 months to setup, and 1 hour to run, but running this model in 2 months and 1 hour, compared with the actual trial which involved thousands of people, millions of dollars, and 7 years? WOW, very very powerful, controverisal for sure, but very powerful. Here’s the link for your reading pleasure. http://www.wired.com/magazine/2009/11/ff_archimedes/…
The Future of Business Intelligence
January 25, 2010
January 2010 Cover
Have you heard of Technology First? Technology First is a Dayton, Ohio based industry-led, industry-driven trade association dedicated to:
- Proactively Representing IT in the Region
- Increasing understanding of Technology First and its value
- Recognizing and promoting our membership
- Highlighting niche technology companies
Technology First looks to strengthen technology thought leadership by inspiring innovation, focusing on new ideas and best practices, presenting leading edge industry information that is both strategic to business and technical folks. They also look to inspire volunteer leadership by encouraging stronger member participation which involves more working committees and develops programming to best meet industry needs. Additionally, they look to engage in conversations with technology community by leveraging interactive social media.
I was asked to prepare an article on the Future of Business Intelligence. Imagine my surprise when that article was selected as their cover story this month! Click here to read. I’d love to get your thoughts.
Have a great week!
- Jodie
If I Had A Hammer…
January 14, 2010
If I had a hammer…
No not the song… There is a story that the IT people like to tell, not sure if it is true but I love it so well…sorry Jimmy B. It goes something like this.
A manufacturing company with a complex assembly line had a machine break down on them. The machine was critical in the production of their products, yet try as they might to fix it themselves, they just could not keep it running 24×7. Pridefully, the plant manager didn’t want to admit that his team couldn’t solve the problem, but he knew that soon enough the company’s product yield would be impacted and someone way above his pay grade would notice. Time to call an expert.
The following week, the expert arrives at the plant. The plant manager escorted him to the offending machine. The expert set down his briefcase and began to ask a few questions of the plant manager and the line supervisor. He then walked around the machine, climbed up the maintenance ladder looking around. Climbing back down the ladder, he asked the line supervisor if he had a hammer. The supervisor looked at him sideways and said, “well, uh, yea, I got one.” So the supervisor went to his toolbox, retrieved a well worn ball-peen hammer and handed it to the expert. The expert climbed back up the maintenance ladder and leaned over the side of the ladder to reach the broken machine. He swing the hammer down sharply with a loud “bang”. Instantly, the machine began to whir, the indicator panel on the side of the machine lit up with all green lights and production was running again!
The plant manager and line supervisor thanked the expert for his help to which the expert replied that he’d send his invoice for services later that week.
The invoice arrived on the plant manager’s desk and when he opened it the invoice contained a single line item for services.
- Repair of Machine…………………………………………………………………………………………………….$10,000.00
The plant manager was not happy. He thought to himself, “How in the world can that guy charge me ten grand for swinging a hammer?” He immediately called the expert and asked him for a detailed invoice. The expert told him he’d send out another invoice immediately. Two days later the invoice arrived. The plant manager tore open the envelope. The invoice read:
- Use of Hammer………………………………………………………………………………………………………..……….$1.00
- Knowing where to strike hammer………………………………………………………………………………$9,999.00
Isn’t this story much like business today when it comes to knowledge? Many companies are now measuring their enterprise data storage in petabytes. Yet with all that data, they still struggle to answer questions such as—Who’s my most profitable customer? Or, Who’s my most in-need customer? Or, which customer is likely to leave for my competition? How can I increase my business? Where should I focus my efforts? The answers are very likely embedded deep in the data stores of the company but the decision makers can’t get the answers they need, when they need them, how they need them, and how to apply the answers. And therefore they aren’t getting the knowledge they need. They have the “hammers” but they aren’t helping. Enter Business Intelligence. Sure, BI has been around for a long time, but it’s evolving just as today’s businesses are. In today’s world, you need more than data. You need more than information. What you need is knowledge. The fluid, meaningful, applicable evolution of data that allows you to “fix your broken machine”. BI is your answer to unlocking the knowledge you need.
If you’re asking yourself important questions to which you have no answers, might be time to call the expert.
The Phrase Business Intelligence
January 5, 2010
I first came across the word “Business Intelligence” at the 1999 Cognos meeting in Toronto when their CEO announced the “new IT category” as part of the leadership strategy. Their marketing gurus must not have done a manual search or focus group since there wasn’t any indication that anyone really knew why it’s called “business intelligence.” Let’s look at the historical words in this category of making data more meaningful. Throughout my 29-year career, Information Technology Professionals have tended to over-complicate what they are trying to accomplish by coming up with descriptive labels that tend to remind me of a NASA space mission. Back in the Eighties, we called it Decision Support Systems (“DSS”). In the early Nineties, it was referred to as Executive Information Systems (“EIS”). Then, with the explosion of relational data base technology, the new movement became coined as various tangible models: Data Warehousing, Data Marts, Closets, Data Mining, and the like.
From an IT perspective, there are a lot of differences between these definitions throughout the years. At the same time, how do they really differ from a business executive viewpoint? Are the decisions in business being made today differ significantly from decisions that were made yesterday? Does the thinking process differ from an analytical viewpoint? Does having more data mean that you can make better decisions? Are decision-makers better off with all of the data that is available? How does the business executive think about “business intelligence” from an information viewpoint?
Here’s a three-part “Maslow’s Triangle” layered model to think about Business Intelligence from a business perspective.
1. How’s Business?
First, at the base of the triangle, you have to ask “How’s Business?” This layer really emphasizes “time over periods” of transactions. Traditionally, this area is termed “transactional reporting” and simply put, is giving the user their numerical tabulation of data at the end of a period. What would a business person define as “Best in Class” in this area? Give me my reports as near-time as possible for the period that I am looking at and be able to sub-category my different business lines, product lines, financial divisions, etc. Most of the data could be described as the data from “double-entry bookkeeping systems.” With today’s ERP-style systems, this kind of information is fairly accessible as long as you are dependent on internal data only. Some data feeds may be external feeds or internal non-structured data sources that still have the same timeframe. Examples would include “customer satisfaction” data, quality data, and other operational inputs.
For example:
“What is the revenue over the last quarter?”
“How many X was sold in the last week?”
“What is the profit for the month?”
2. Why?
The next layer up the triangle is simply put “Why?” Why did the business’ Eastern Region have a 5% increase in sales year over year? Why did we miss our numbers in the last week of the quarter? Why did our market share grow in our mature product line in the last 2 quarters?
3. What If?
The last layer of the triangle is “What If?” If one can receive their business results from “How’s Business” and then is able to determine “Why” the business performed in this fashion, the “What If?” pinnacle of the triangle will provide a roadmap for the decision-maker to model their potential decisions that they have in mind.
For example, if one knows their financial performance and also sees where the business over-achieved and where it under-achieved, it is able to move resources of the business (management & money) to the area of need. Whether the strategy is to provide more or less resource is up to the person involved. The numbers themselves are not going to make the decision.
So, then is “Business Intelligence” an oversold phrase in the world of Information Technology? A “Qualified Yes” and a “Qualified No.” The challenge today is that the tools actually work and work well if the approach taken is right. At the same time, recent publications and noted experts all agree that the road to Business Intelligence is cluttered with a lot of failed attempts, a lot of capital spend that isn’t going to be realized from an investment viewpoint, and a lot of disenfranchised users.
I’ll write about this dilemma in my next blog.
The VLOOKUP Hookup
December 22, 2009
Companies invest large amounts of money, time, and other resources acquiring and implementing reporting and analysis software. I’ve seen organizations invest hundreds of thousands of dollars in projects and fail to realize a decent return on their investments. The point of this series of posts is to educate you about the reporting and analysis capabilities of a tool your organization probably already owns: Microsoft Excel.
In this series of posts, I will discuss a number of these capabilities and will give some concrete examples of how to utilize them.
I will be using Excel 2007 for these examples. Much of this functionality is also available in Excel 2005, it’s just not as easy to use and does not have some of the more advanced features.
Getting the Data
The first step in any effort is to get some data into Excel. We’ll start out using a simple static list. You probably already use lists like this regularly. If you don’t utilize Excel in this way today, think of the reports that you work with from the various systems that you run your organization with. In most cases, you could probably either copy andhttp://thefuturevalueofbusiness.com/wp-admin/post.php?action=edit&post=745&message=6 paste or import these reports into Excel to get some data to work with.
In future posts, we’ll cover a much more powerful method of acquiring data by connecting to external databases from within Excel. For now though, we’ll stick with this simple example.
I’ll be working with the sample database that comes with Microsoft’s SQL Server database software. This sample database contains information about a fictitious company called Adventure Works. Below, you can see that I have an extract of order information that I’ve pasted into Excel.
This is the most common manner in which people utilize Excel for reporting purposes: simple lists of data pasted or imported from other sources. In most situations, this data comes from existing reports or queries. My example above is a very simple query…you can see that we don’t even have names or descriptions for most of the data. For example, Column F is showing us the Product ID instead of the Product Name.
The best way to solve this problem is to have the author of the report or query modify it to include the Product Name in addition to the Product ID. Let’s imagine that this is not a realistic option though; there is a way that we can solve this problem using an extremely powerful Excel formula called VLOOKUP.
Using VLOOKUP
To expand on our situation above, let’s imagine that I have a second worksheet in my Excel workbook. I have an image of this second sheet below.
The Product ID in column A corresponds to the Product ID in column F on the Orders List. We are going to use VLOOKUP to take the Product ID in the orders list and lookup the Product Name in the product list.
To make the formulas a little more understandable, I am going to rename the Sheet with the order list “Orders” and I am going to rename the Sheet with the products list “Products”.
On the “Orders” sheet, let’s insert a column immediately to the right of the Product ID. We’ll label it “Product Desc” in Row 1. In Row 2, we’ll enter the VLOOKUP formula:
| =VLOOKUP(F2,Products!A:B,2,FALSE) |
The parameters (the information between the parentheses) tell Excel how to lookup the value we want:
- The first parameter, “F2”, tells Excel what value we are performing the lookup for. In this case, we are looking up the Product ID.
- The second parameter, “Products!A:B”, tells Excel where to go to do the lookup. Here I selected the first 2 entire columns on the “Products” sheet.
- The third parameter, “2”, tells Excel to bring back the data in column 2 from the lookup list when it finds a match for the value from cell F2. I know that’s a confusing sentence at best, but it will make sense in a moment.
- The last parameter, “FALSE”, tells Excel that we want it to return only an exact match for the value we are looking for. If Excel cannot find an exact match for the Product ID, it will return an error indicator.
The Results
Now, let’s take a look at the results of our formula. The screenshot below shows what I have now.
This screenshot shows a few rows from the “Products” sheet:
Hopefully you can see how VLOOKUP works now. Excel took the value in F2 in the “Orders” sheet, 776. It went to the first column of the range we gave it; that range was Columns A and B of the “Products” sheet. It scanned through that range until it found a match for 776. It then took the value in the 2nd column of the range, column B, in the same row and returned the value in that cell (“Mountain-100 Black, 42”).
One thing I didn’t make clear before that I want to point out now. VLOOKUP is always going to look in the first column of the lookup range for the matching value. In our example, the lookup range was columns A and B of the “Products” sheet, so Excel looked in column A for the matching value. There is no way to tell the formula to look anywhere other than the first column; so you either need to cut and paste the columns to get the right one first, or just change the reference so the lookup column is first.
To complete our list, we can just fill down the VLOOKUP formula in column G to the bottom of our orders list. Now we can analyze our order data with actual Product Names instead of just Product IDs.
Summary
VLOOKUP is useful in many other situations…you can probably imagine a few other uses for it yourself. It is very handy to use it as we did in this example though. Even though we could have accomplished the same goal by having someone in IT modify the query or report, now you can be a little more self sufficient with your reporting needs.
In my next post, I’ll cover a few more features like filtering and date manipulation. Ultimately, we’ll move on to Pivot Tables and External Queries which provide very powerful mechanisms for analyzing data and can compete with some features offered in expensive reporting software.