The main entities of Data Vault are Hub, Link and Satellite.

HUB Entity (HUB_): This is a defining entity. It contains a unique list of business keys. These are the keys that businesses utilize in everyday operations. For example, employee number, SSN number, Product Code. So the attributes of HUB are:

• Surrogate Key – This is a Primary Key of hub and holds 1-to-1 relationship with the Business Keys.
• Business Key – This is a Primary Key of the source system. This can be a composite key. ETL checks this key’s existence in the hub table and inserts one if it doesn’t exist.
• Load Date Time – The datetime of the key / record when it was first loaded into the table.
• Record Source – The name of the source the record originated from. This is useful for data traceability.
• Record Begin Date Time – The datetime when the record became active in the source (if available) or the datetime when ETL has been run.
• Record End Date Time – The datetime when the record is closed. This can only be detected if the logical deletes are supplied or derived in some manner.

LINK Entity (LINK_): LINKS are constructed once all the HUBS are identified. Links are relationship entities.  These are the physical representation of m-to-m 3NF relationship. It represents the relationship or transaction between hubs. The link table contains the unique list of relationships between hub keys. When a relationship arrives, it simply gets loaded into the table if doesn’t exist. Typically, the link tables translate into fact tables in the datamart access layer. For example, the link between employee number and the project number. The other attributes of LINK are:

• Surrogate Key – This is a Primary Key of the table and is useful when a link contains more than two hub keys as composite key might cause performance problems. This is also
  useful when the granularity of the link changes (a hub key is added) or history needs to be maintained on the relationships.
• Hub Key 1 to Hub Key N – The surrogate keys from the hub tables that are involved in the relationship.
• Load Date Time- The datetime when the record was loaded into the table.
• Record Source – The source system name from where the record or relationship was loaded from.

SAT Entity (SAT_): SATS holds descriptive information about the hub keys or the relationships. The satellite is most closely resembles Type 2 Dimension. When the data changes, a delta record is inserted into the table and if the certain columns changes faster than others then these can be split into two different tables to avoid data replication. For example, employee details such as employee name, address, phone number, email address in the satellite off of hub  or time spent by an employee on a certain project in satellite off of LINK that stores the relationship between an employees and projects. The other attributes of SAT are:

• Hub or Link Surrogate Key from HUB or LINK table. This is part of the primary key.
• Load Date Time – The datetime when the record was inserted into the table. This is part of the primary key.
• Surrogate Key – This is optional. It is useful when satellites have multiple values such as multiple home addresses.
• Record Source – The name of the source.
• Record Begin Date Time – The datetime when the record became active in the source (if known) or the datetime when ETL has been run.
• Record End Date Time – The datetime when the record is closed.

And stand-alone tables such as calendars, time, code and description tables may be used.

Below is a snippet of a Data Vault Model housing borrowers who have taken out Student Loans:

Modeling Rules for Each Part of the Entity:

FOR HUBS:

• Hubs keys cannot migrate into other hubs (no parent/child like HUBS).
• Hubs must be connected through links.
• More than two hubs can be connected through links.
• Surrogate keys may be used.
• Business keys are 1 to 1 relationship with surrogate keys.
• Hubs primary keys always migrate outward.
• Hub business keys and primary keys never change.
• If a hub has two or more satellites, then a point-in-time table can be built for ease of joins.
• An ‘UNKNOWN’ business key record can be inserted into Hub that can be used to tie other data in links and sats that has no business keys in source. This kind of data is usually a bad/incomplete source data.

FOR LINKS:

• Links can be connected to other links.
• Links must have atleast two hubs associated with them in order to instantiated.
• Surrogate keys may be used.
• The combination of surrogate business keys made a unique key.
• Does not contain descriptive data.
• Does not contain begin and end dates.

FOR SATS:

• Satellites may be connected to hubs or links.
• Have 1 and only 1 parent table.
• Satellites always contain either a load date-time stamp, or a numeric reference to a stand-alone load date-time sequence table.
• Primary key is a combination of ‘surrogate key’ from either hub or link and the load datetime stamp.
• Surrogate keys may not be used.
• Must have a Load End Date to indicate when the CHANGE to the data set has occurred.
• Satellites are always delta driven. Duplicate rows should not appear.
• Data is separated into satellite structures based on 1) type of information 2) rate of change.

DV model utilize bits of both 3^rd Normal Form and Dimension Modeling concepts.  This approach has made the model simple, flexible, expandable, adaptable and consistent.

• Adapted many-to-many physical relationship structure from 3NF that became a LINK table.
• The LINK table is also similar to factless fact in Start Schema.
• Adapted the notion of 1 to 1 (business key to surrogate key) tracking from dimensional modeling (type 1 dimension).
• Adapted the notion of “data over time in a separate table/structure” from dimensional modeling (type 2 dimension). This resulted in a SAT table however it is fundamentally
  different, in that it is a child dependent table, whereas the dimension is a parent table to the facts.

This is it for now. In next post(s) we will look into some examples which will show how Data Vault technique overcomes the limitations of 3NF and Dimensional Model structures when applied as an Enterprise Data Warehouse.

- Jyothi

Source: tdan.com, danlinstedt.com

The first entry is the slide deck from the recent Advanced Architecture Conference in Denver: http://danlinstedt.com/datavaultcat/datavault-advanced-architecture-conference-slides/

There is a wealth of information contained herein that outlines at a high level the benefits and reasons why the data vault method is enabling pervasive BI. It goes into details about comparing 3rd normal form and the star schema form to the value the data vault brings to the table. It outlines the components of a data vault, including the hubs, links, satellites, PIT tables, and bridge tables.

It was nice to see the connection with the methodology to emerging tools. There are now more than a handful of tools that are undertaking the task of automating the ETL and loading processes of an EDW. The method of the data vault allows for a lot of the technical innovation. Dan presents the case well that businesses today want answers faster and cheaper today as well as having the system flexible enough to evolve with the business over time. Most EDW systems today fall short of being pervasive, and often that is a methodology problem that appears to have an answer.

In addition to these topics, Dan pushed forth the proposition of an “Operational Data Vault” and began to lay out how this connects to Business Intelligence. This section was very interesting and helped to turn on a few light bulbs and progressive thoughts. I love the idea of pushing the intelligence as far as possible while still keeping the style and benefits. I agree with Dan that this is still so new that there are not yet any vendor applications or tools that can help with this…yet.

The presentation does leave a little bit of gap with the more advanced concepts, such as change management and how to leverage a data vault using agile principles. It is also a bit light on how to drive a project plan as well as what to watch out for while attempting to build a data vault style EDW. All in all, he presents a very compelling case as to why the data vault should be strongly considered when delivering Business Intelligence.

The second entry is a public rebuttal to a challenge about the validity of the data vault methodology:
http://danlinstedt.com/datavaultcat/datavault-new-response-to-frank-habers-part-1/

As Dan states, he felt the need to lay out the facts and correct misconceptions from the original author. After you get past the first few sections of disclaimers and background, the article does a great job of comparing and contrasting methodologies and technical styles for building the EDW.

I often see HDSA (persistent historical data storage area) implemented at client sites and they sometimes go by different names (ODS, staging, etc.). I have yet to see a better discussion of the merits of a data vault compared to the HDSA as he presents here. If you are in an environment that has an ODS or persistent history area, please take a moment to read this section, because the warnings and issues are real and one should not make decisions here lightly because of the long term ramifications.

Dan also goes into great detail to itemize the benefits and value proposition of why you take the steps to have a data vault at the core of the EDW.

I would say that the overall tone of the article still makes it a bit hard to read, but if you can overlook the defensive stance, you will find many gems that will help solidify and justify the purpose and value of the data vault compared to any other technique.

BI systems need DATA and every business has terabytes of real data which can provide them with the information and knowledge they need to make the right decisions on time. But the key is to turn that data into information in a timely, efficient and effective manner once the WHAT AND WHY questions are answered i.e., what information is needed, what matters and why that is required.  In today’s market, every business is in a RACE. The race to conquer others. The race to generate more gains/profits. The race to foresee the risks early on so that they can be avoided.  So time is of the essence here.

An optimized BI system integrates large volume of external and internal near real time data to allow management to create opportunities by making intelligent decisions after performing predictive analysis of their approach on the business. A good BI System is like a GPS. An effective GPS is one that not only shows you a route to your destination but also guides you when you hit roadblock, gives up-to-date external conditions (constructions / traffic) information, provides multiple routes to choose from, suggests you with alternatives for shorter and fastest routes, predict the total time based on your driving behavior, tells you what to expect next etc. Just knowing the path to your destination is not sufficient. You need to know many other factors during the whole ride to reach destination on time and without any hurdles.

For a good integrated BI system, a good Data warehouse architecture needs to be in place.  Data warehouse architecture is “an integrated set of products that enable the extraction and transformation of operational data to be loaded into a database for end-user analysis and reporting”. Below are the pictorial representations of different “flavors” of DW architectures.

Methodologies used by different architecture:

Kimball’s DW Architecture – Is based on ‘Bottom-UP’ methodology.

Inmon’s DW Architecture – Is based on ‘Top-Down’ methodology.

Dan Lindstedt’s Data Vault DW Architecture – Is based on ‘HYBRID DESIGN’

The first two design methods have some limitations for Data Warehouse layer such as inflexibility and unresponsiveness to the changing departmental needs during the implementation phase, insufficient auditability of data back to its source system, inability to integrate unstructured data, inability to rapidly respond to changes (organizational changes, new ERP implementations) or difficult to load type 2 dimensions in real time. This is where DATA VAULT came in to rescue. Data Vault follows a ‘HYBRID DESIGN’ methodology which follows ‘TOP-DOWN ARCHITECTURE WITH A BOTTOM-UP DESIGN’.

The model is a mix of normalized modeling components with type 2 dimensional properties. In this model, the DW serves as a backend system that houses historical data which is integrated by the business keys. All data ‘good, bad, incomplete’ gets loaded into the data vault and all the cleansing and application of business rules takes place downstream i.e., out of DW. This means that Data Vault model is geared to be strictly a data warehouse layer, not as a data delivery layer which still requires physical or Virtual star schemas or cubes for Business Users or BI tools to access.

Bill Inmon in 2008 stated that the “Data Vault is the optimal approach for modeling the EDW in the DW2.0 framework.”

In Part 2 and 3, I am going to explain different components of Data Vault and it’s power with the help of some examples.  That will clearly explains why the Data Vault should be a preferred “flavor” for different businesses.

- Jyothi Kaparthi

The way to approach testing a Business Intelligence (BI) system is to get the business to have ownership and buy-in early and often. The business users should be writing test cases and be responsible for executing them from a business perspective as well as trains them on the content in the system. The technical people should be ready to assist with query development or whatever to help the testing to be completed.

There should be some validation that is part of the design of the Extract, Transform, and Load (ETL) process itself. Some of this is to make sure mechanically things happened as they should and that there are appropriate logs when they don’t. In addition, the ETL developers should perform some kind of UFI (Unit Function) testing prior to moving to a TEST environment as well as a code review or peer review. Depending on the complexity of the ETL process, one generally don’t test each component of the process due to the details involved, but focus more on the net result of the test (i.e., all rows were inserted with no errors and all columns contain values – what happened in between is not as important to test because the load was successful).

In addition, the technician should take the next step of developing quality controls that make sure what was in the final table structures is what was expected. For example, have a report from the Operational Data Store (ODS) area that groups and sums some business keys with some key metrics and compared them to the results from the new implementation area, while highlighting only variances. This should be sent to a data governance team every morning. As long as it was clean, the BI team is sure mechanically things were working pretty well.

Depending on some of the business rules implemented, one may need to have reports that highlight “Unknown” values and other things that need to be dealt with by the business. Some of these scenarios should become test cases. The business should be using the Ad-hoc environment to be writing reports and queries to test the results. Ultimately, these reports should be reviewed by the data stewards as part of the data governance initiative.

For the most thorough results as well as the highest quality BI environment, everywhere there was a business rule implemented, there should be a test case that verifies the rule was implemented correctly. Depending on the volume and complexity, one may need to prioritize them and tackle the most important ones first.

For the documentation, it can be as simple as keeping a spreadsheet with the following items:
• Test Case #
• Test Case Description
• Tester
• Date Tested
• Expected Outcome
• Actual Outcome
• Pass/Fail

It is critical, as originally stated, to get the business users involved in the testing of the deliverables. There have been cases where the business thought what they were using to compare balances with was correct, but were eventually convinced the BI application was correct and they had a broken business process instead. This is most difficult because it is on a case by case basis, but this usually becomes the biggest challenge and hurdle that needs to be overcome to be perceived as successful. Ultimately the business must provide you with the information to know whether “the values put into the Data Warehouse or BI dashboard are correct”. And you are completely dependent on the business rules they gave you are correct (a lot of times they aren’t in version 1)…and even more risky if there is no data governance process in place.

A word of caution, if you don’t get the business buy-in on testing, they will certainly blame you when things aren’t correct in production (especially if that happens for things that were overlooked in testing). It is wise to have a step where the business had to sign off on testing and that they were comfortable with what was moving into production and that was very helpful when issues arose. Because they were involved in the process and it was not mostly IT doing the testing, finger pointing was kept to a minimum. In addition, shared success and teamwork was fostered bridging the gap between business users and Information Technology (IT) groups that sadly exists in a lot of organizations.

Well, BI in the cloud just made a big leap forward.  The poster child of cloud companies, Salesforce.com, has now entered the mix.  Salesforce announced a few days ago the availability of www.database.com, a service they (salesforce) will be offering, based upon their existing architecture.  WOW…  The game should start moving even faster now, what with Salesforce now directly competing with the major Database vendors (Oracle and Microsoft specifically).  I personally find it interesting as well that of all the areas Salesforce could have brought to the cloud (vertical/horizontal solutions, ERP/MRP systems, yet another email system, etc) they chose Database.   Oh, don’t get me wrong, it makes sense relative to leveraging their existing infrastructure, but I think it’s an interesting vote on where BI, Data and the cloud are headed.

• What are my customer’s buying is a basic question, but moreover, what products do they buy together?
• Which products do they buy when times are tough?
• What did they buy during the last recovery?
• What aren’t they buying, and what should I recommend they buy?

All great questions, and clearly a value add of a strong BI platform.

eWeek – Using-Business-Intelligence-to-Find-Your-Economic-Recovery

We were Tweeting during the event.  Here are the nuggets of info we learned from Steve and Dennis:

Not all of their slides could be shared due to the confidential nature of some of the dashboards, but here is the rest of their presentation.

Enjoy!

- Jodie

You might be thinking to yourself, “Yes, but it’s too difficult to get access to each system and pull the data together.” Or “Oh, I have to get our corporate IT department involved and I don’t have time to wrestle with the “process.”

You need to make critical business decisions fast and need an easier way?

YourView – The Vision
LUCRUM has a deep history working with data.  We had a vision to create a product that helps bring data together through an easy and intuitive application. As we started looking at the problem from a different angle, we found that the solution is less about data and more about combining data to help answer “top-of-mind” business questions. Imagine viewing on one report your forecasted sales data and customer purchase history. You can start answering question like: 

• What is my expected versus realized revenue gap?
• Are my sales reps properly aligned?
• In what industries are my biggest customers?
• What else could I be selling them?
• What are my realized margins per sales person?

 The answers to your questions maybe sitting right on your own computer. It could be in a report or in the various Excel spreadsheets you use everyday. If you can access the data, then YourView can help you gain a better understanding.

Solution
YourView allows data to be combined from multiple data sources using a very simple application like Microsoft Excel. Most line of business applications allow data to be sourced to Excel through a reporting or export feature. In addition, YourView allows you to source data directly from cloud services (salesforce.com). Once the data is inside YourView, each different source can be combined together into a single view utilizing a common business entity such as customer name. Combining the data is performed through YourView’s simple Business Entity Mapper feature. The data is then loaded into a relational data mart, which can be used to seek answer to the “top-of-mind” questions about your business.

 Specifications
YourView is a Microsoft Windows desktop application, which allows multiple data sources to be loaded, categorized and mapped, and then loaded into a Microsoft SQL Server database. Data is modeled using the Business Entity Mapper, which defines the business categories and measures. YourView will create a physical database and will load the data into the database. YourView utilizes LUCRUM’s dynamic data loader toolkit (DDLT) as the ELT (Extraction, Loading, and Transformation) engine.

 Features:

• Load data from Microsoft Excel 2003
• Connect directly to salesforce.com and pull data into YourView (requires an authorized SFDC account with API access)
• Edit the imported data directly inside YourView
• Classify data as business dimensions (categories) and facts (measures)
• Converts data into SQL Server Types: varchar, nvarchar, decimal, integer, and bit
• Performs data validation to help find and determine data integrity problems
• Creates a physical database and loads the data based on the model generated from the Business Entity Mapper

 Requirements:

• Microsoft Windows XP SP3/Vista/Windows 7
• Microsoft.NET 3.5 SP1
• Microsoft SQL Server 2005/2008 Standard Edition or Express Edition
• 1 GHz Processor or Higher
• 400 MB RAM
• 10MB of Hard Disk Space for product installation – additional disk space is required for the YourView deployed data mart

OLAP and Performance Feedback

Improvements to organizational effectiveness can also be realized by utilizing OLAP tools to provide performance feedback to individual employees.  Improved performance feedback will help employees achieve group and individual performance objectives.  Increased attainment of these individual and group performance objectives will, with proper alignment of these objectives and organizational objectives, improve organizational effectiveness.

There are several advantages to providing performance feedback with an OLAP tool.  If the situation is right, feedback can be provided:

• At an individual level
• On a larger sample of employee activity
• Quickly
• In a meaningful manner.

Common Problems with Performance Feedback

Organizations often make attempts to improve the provision of feedback to employees.  Newsletters with departmental performance numbers, posters in gathering places displaying performance charts, and managerial reports with quantitative measures of performance are all attempts to improve the distribution of feedback to employees throughout the organization.  One problem with such efforts is that they are usually not provided at an individual level.  Feedback on departmental, team, or group performance is certainly helpful but depending on the size of the group, its effect will be limited.  Individual performance feedback has its own problem in that it is often time prohibitive to provide extensive individual performance feedback.  The result is often weekly or monthly group performance feedback with individual feedback coming only during annual or quarterly reviews.

Individual performance reviews often suffer from another problem: small sample sizes for review.  If an insurance company is reviewing the performance of claims adjusters using manually prepared data, it may be impossible to review more than a small sample of the adjuster’s work over what is typically a long review period.  Small samples may, of course, result in a flawed appraisal of an employee’s overall performance.

The elapsed time between events reviewed and performance appraisals is also a problem with traditional feedback provision.  Consider the timing of typical reviews: an employee makes a mistake in handling a situation in January, the incident turns up in a sample taken in May, and a review is finally conducted in June.  If a review had been conducted immediately following the incident, the chance of the employee repeating the mistake will obviously be lower.

Traditional feedback provision often suffers from poor presentation of the message.  An interview conducted by a busy manager attempting to perform a number of appraisals in addition to other work may not be optimally effective.

Performance Feedback Improvements with OLAP

Utilizing an OLAP tool may remedy some of the traditional problems with employee feedback.  Imagine again the situation of an insurance company reviewing the performance of claims adjusters.  As a solution to the problems listed above, an OLAP cube could be developed and made available to adjusters on a daily basis.  Adjusters could be presented with individual performance feedback delivered via the web.  They could see at a glance how their activity the previous day compared to group averages and organizational objectives.  Exceptions could be noted immediately by the individual employee, rather than organizational objectives.  Exceptions could be noted immediately by the individual employee, rather than a manager, and quickly corrected.  Feedback could be provided on all activity from the previous day or week rather than on a small, dated sample.  Finally, feedback could be presented in easy to understand charts which, in addition, roll-up to display departmental and organizational performance as well.

Improved performance feedback gives employees the ability to monitor their own performance and to take corrective action quickly.  By improving the ability of individual employees to meet their performance objectives, the ability of the organization to meet its objectives and fulfill its mission is improved as well.

Conclusion

OLAP technology can improve organizational effectiveness by:

• Improving management’s knowledge of progress on objectives
• Improving employee coordination on efforts to achieve these objectives
• Communicating the link between employee effort and performance
• Communicating the link between employee performance and reward
• Improving employee performance feedback.

Although OLAP tools can provide assistance in these areas, their impact is obviously limited by factors specific to each organization.  An OLAP tool cannot compensate for poor development of objectives, poor performance reward systems, or any of the other organizational factors discussed.  Utilizing an OLAP tool as I’ve described in this series with no attention given to the underlying systems it is trying to address will, at best, have no effect.

In an organization that has clearly defined its objectives and has implemented well-designed reward systems, utilizing an OLAP tool as we’ve discussed can offer a tremendous payoff.  The ability to provide employees with improved performance feedback and to demonstrate the link between individual performance and organizational performance is extremely valuable.  By helping an organization align individual goals with corporate goals, an OLAP tool can help an organization become more effective.

OLAP’s Impact on Organizational Effectiveness

How can an OLAP tool help improve an organization’s performance as measured against its objectives?  Answering this question requires a greater understanding of how strategies and tactics are implemented within organizations.  I’ll use a model of organizational effectiveness developed by Michael Beer to illustrate the implementation of strategies and tactics.

The picture below shows a simplified version of a model of organizational effectiveness developed by Michael Beer (Note on Organizational Effectiveness, 10).  Business goals and strategy influence and are influenced by top management.  Management determines and implements the proper organizational design to achieve the organization’s goals.  The design of the organization, in turn, influences human resources attributes of the organization.  Finally, these HR attributes directly impact organizational effectiveness.

This simplified version of Michael Beer’s model is presented again below.  Added to the model though, is the position of an OLAP tool in improving organizational effectiveness.  OLAP technology exerts its influence on organizational effectiveness in three sections of the model:

• Management
• The Measurement and Reward Systems aspects of Organizational Design
• The Coordination aspects of Human Resources.

While the impact of OLAP technology in each of the areas above is slightly different, each is related and shares a common trait: improvement in communication.  Utilizing OLAP tools to improve communication requires a broad audience for their utilization.  OLAP tools are traditionally utilized by analysts and managers.  In this model, front-line employees become critical users of the tool as well.  The wide-scale availability of web-based OLAP tools makes such organization-wide implementations cost-effective.