Data Auditing is the process of ensuring data quality from the beginning of the BI process to the final destination in a repeatable and measured way.   This includes validation of data that arrive from source to staging to star to cube (if exists).  Where ever business logic can be implemented, a data audit can be used to make sure the quality of the data is consistent. 

One example of data auditing I have used was for a ‘technical glitch’ with the use of SQL Server Analysis Services (SSAS).  The decision was made to not change source data that was of bad quality and load it into the Kimball star schema as it was.  (Now this decision can be debated, but one major reason this was done was to easily expose data quality issues to the users as they believed there would be an inconceivably small amount of data quality issues.) What was discovered with using SSAS is it would not handle dates with a year prior to around 1500.  Some of the dates in the source system had the year 200 instead of 2000 and so on.  A Data Audit routine was designed to look for these dates into the stage tables and change them to a pre-determined default date. This allowed the SSAS job to complete and the cube to process.

This process seems much like any other data quality processes.  The true auditing came from how these instances (and the others) were reported.  The code developed to catch the data quality issues also entered data into  “performance” stars schemas designed to provide data on the nightly process.  This data was then shown by a dashboard used by the internal IT, BI staff.  Every morning we could see how many rows of data were caught by each audit.  This allowed us to make quick decisions on how to handle the data to change in the source system and even change in the data warehouse (including staging tables to make sure there was not a type 2 slowly changing dimension row added by the change to the original source data).

There were other data audits that summed specific counts of rows from the source system and made sure that count was the same in the star and cube due to how the star was loaded.  Another data audit allowed us to show the measures in the fact tables and the cube were the same based on logic used.  These types of data audits are not used to catch data quality, but to explicitly show that on a day-to-day basis, the results are the same.  This audit was extremely necessary to build confidence in the data.  Confidence in “the numbers” was extremely important to our clients and this was the simplest way for us to convince our end users they were getting what they were supposed to get. 

The best way to implement a data auditing solution is to use the existing BI tools to build the report, dashboard or any other means to expose the audit.  Even the simple use of Excel against the cube and star can be used.  Any way that can be easily maintained is prefered.  Of course the data needs to be understood in order to make sure this is done correctly.  If data is distributed in the star to form a lower level of granularity, then it needs to be summed back to the original level and compared to what is in the source.  This could mean there are rounding errors present, but that should only provide around a penny difference. 

A problem with Data Auditing is the trade-off of time developing the data auditing process takes away from time to develop the business needed parts of the BI solution.  One way to incorporate this is to build it into each project.  The first project to utilize the concept of data auditing will take longer due to the need to build the underlying data structure and processes.  Once this is started and built, the next project to use the structure will take less time to ‘plug in’ just like the idea of re-using conformed dimensions.

Data Auditing’s benefits provide a repeatable way to show data throughout the entire BI process is correct.  This concept is part of a good data quality/data governance solution.  The ability to ‘watch’ the data as it goes through the entire BI process to make sure it means what it supposed to mean will provide a security blanket for the end users.  The end users can say “This data is from the data warehouse and I am positively sure it is correct.”  How much value to the business and reassurance to IT does that grant?

– Doug

LÛCRUM specializes in Business Intelligence (BI) solutions. We develop BI solutions in an iterative, rapid development environment that provides a competitive advantage. We create information that is accurate, actionable, complete, secure and timely – revealing the future value of your business.

Consultants at LÛCRUM have the opportunity to grow their skills and level of consultancy as an employee. We offer additional training and certifications allowing you to expand your technical background. In addition, there is the opportunity to work on a variety of projects with numerous clients and the ability to earn bonuses throughout the year solely based on your own contributions.

LÛCRUM is currently seeking consulting candidates that can work at one of the following levels:

• Consultant (0-3 years consulting experience)
• Sr. Consultant (3-8 years consulting experience)
• Principal Consultant (8 years plus consulting experience)
• Associate Partner (10 years plus consulting experience)

CONSULTANT
LÛCRUM Consultants typically possess from zero to three years of consulting experience. Consultants are highly motivated Information Technology professionals who are actively developing their technical skills. Consultants work within a team environment, follow the lead of more senior consultants, and interact frequent with LÛCRUM and client managers. Consultants:
• Possess a depth of knowledge in one or two core technical skill sets and continue to broaden their skill sets.
• Gather and analyze client requirements.
• Assist in design, development, and delivery of Business Intelligence and Data Warehousing solutions for LÛCRUM’s clients.
• Perform coding, testing, and configuration changes as directed.
• Design technology-based solutions to meet functional and technical requirements.
• Provide project management support.
• Create client documentation and assist in helping clients to learn and use LÛCRUM technology solutions.
• Complete projects on-time and within budget.

SENIOR CONSULTANT
Senior Consultants typically possess at least three years of consulting experience. Senior Consultants are accomplished Information Technology professionals, having deep technical expertise which they continually broaden. Senior Consultants require minimal supervision and are capable of leading small teams. Senior Consultants:
• Possess deep experience in one or two core technical skill sets and continue to broaden their skill sets.
• Respond professionally and thoroughly to client demands.
• Apply practical and innovative approaches to problem solving.
• Proactively involve senior members of the team on critical issues, bringing critical issues to management’s attention.
• Willingly provide assistance to others.
• Take initiative to gain leadership experience.
• Attain technical/business certifications.
• Demonstrate an understanding of project risk and scope.
• Produce accurate project estimations.
• Provide timely and accurate status reports.
• Submit qualified leads.

PRINCIPAL CONSULTANTS
Principal Consultants typically possess more than eight years of consulting experience. Principal Consultants have very broad technical expertise as well as architectural capabilities. Principal Consultants provide large and complex project leadership often concentrating on one client account for a period of time. Principal Consultants:
• Possess broad technology expertise and architectural capabilities.
• Are empowered to resolve client/project issues while keeping management appraised.
• Package client solutions for repeatable business.
• Assist in the development of consulting skills in other LÛCRUM consultants.
• Demonstrate account management expertise.
• Proactively resolve client issues and involve management as appropriate.
• Understand the full project life cycle including risk, financials and managing for follow-on engagements.
• Demonstrate effective delegation to others.
• Demonstrate growth in skill breadth.
• Provide leadership within LÛCRUM and to client.
• Demonstrate the ability to lead mid-size project teams.
• Understand the sales process and participate as requested.
• Generate creative solutions appropriate to market conditions and/or client needs.
• Tailor professional communication to audience.
• Submit qualified leads.

ASSOCIATE PARTNER
Associate Partners possess at least ten years of consulting experience. Associate Partners consistently turn leads into revenue generating opportunities while managing multiple accounts with the highest degree of business acumen. Associate Partners:
• Possess broad technology expertise and architectural capabilities.
• Create and manage large and complex budgets.
• Understand financial reports.
• Understand how technology can improve business.
• Seek new business and convert leads into opportunities.
• Submit a large number of qualified business leads that generate a significant amount of revenue.

We welcome you to explore a career opportunity with LÛCRUM. I may be contacted by calling 513-564-6979 or by email – pryan@lucruminc.com

Recruiting Brochure

Current Technologies being sought:

EPIC Clarity Developer/Admin
Required:
- Clarity ETL Support
- EPIC Database Care & Maintenance (learning it well)

Nice to have:
Shared support of Business Objects server deployments
Design of BO Universes and Webi, also some SDK development as needed
SQL Server Administration –Initially Low with capacity / desire to be increased
EPIC KBSQL
Support of EPIC’s Meaningful Use Initiatives
Helping us develop BO Policies & Procedures
Some Xcelsius programming

Sr. Consultant OBIEE Developer
Description:
OBIEE 11.1.1.5 experience
Reports/Dashboards Development
Oracle 11g
Data modeling and building out dimensional/star schema data base structures
Ability to ask good business questions, be engaging and consultative

Consultant Cognos Developer
Tools/Experience needed to perform this role:
Cognos Framework Manager
Cognos Reports Studio
Understanding of data warehousing concepts
Ability to gather/define requirements, design, develop, test and implement reports
Experience training others on the Cognos tools stated above
Ability to work on and off site
Prior consulting experience

Principal Business Objects/Xcelsius Developer
Tools/Experience needed to perform this role:
Business Objects XI
Xcelsius 2008
Develop universes
Dashboard Reports
Requirements gathering with key decision makers and users
Ability to work on and off site
Prior consulting experience

Sr. / Principal Database Developer Lead
SQL 2008
SSIS
SSRS
SSAS
Data Warehouse experience
Requirements gathering
Customer Facing
Some Architecture experience and Project Management skills
Prior consulting experience

Consultant Database Developer
SQL 2008
SSIS
SSRS
SSAS a plus
Data Warehouse experience
Requirements gathering
Customer Facing
Prior consulting experience

There are many design methodologies that take these issues into consideration.  There are advantages and disadvantages to both traditional (and non-traditional) methodologies which I do not cover in this post.  My goal is to bring up points of view of why and when a data warehouse may or may  not be used.  What I would like to cover is:

• The Corporate Information Factory (CIF), based on the Inmon approach
• The Kimball Style of data warehousing
• BI using no data warehouse at all

Corporate Information Factory

The Corporate Information Factory methodology, in a nutshell, says there is no way of getting around this inevitable fact of the need for a data warehouse.  In order to have a successful and sustainable BI program, a data warehouse is needed.  Not only is it needed, it needs to be completely designed, built and populated prior to any further analysis or BI work can be done.  This is due to the nature of how business concepts are intertwined within each other necessitating the big picture view.  This style also views the architecture process more from the IT/data perspective compared to the business need point of view.

Kimball Methodology

The Kimball methodology of data warehouse design is not as structured and regimented as the Corporate Information Factory.  The Kimball data warehouse is the sum of its parts; meaning one area of the business could be designed, developed and deployed providing BI insight while other aspects of the business have not been discussed.  This concept will speed the development of the data warehouse compared to the CIF, but the underlying data warehouse can become much more complex as more and more is added to it along with the possibility of rework.  This style views the architecture process from the business needs point of view compared to the IT/data perspective.

No Data Warehouse?

What about not using a data warehouse?  In the new age of Data as a Service (DaaS), Master Data Management along with Service Oriented Architecture (SOA), why re-store data from disparate systems?  Why not store the metadata of where the data is found and attach the business logic to the SOA call?  This can be a very powerful way to gain insight into data.  The idea that the development of a data warehouse can be done without the data warehouse.  There are already tools that will do this.  One of them is Qlikview from Qliktech.  The basic premise behind this tool is to allow the user to develop the Transform and Load aspects of ETL (Extract Transform and Load) in memory to delivery very quick analytics in a solid visual manner.  This tool is not a methodology, but SOA could be used in a larger context with the same principles. This style views the architecture process as something the business could do, but IT does not have to do.

The idea that a data warehouse is necessary for a successful BI implementation is not necessarily true.  A data warehouse is not necessary to have analytics or provide a picture of the data you have.  I believe it is very questionable to say this process is sustainable to leverage every benefit for BI.  The very important aspect of BI that cannot be overcome by SOA, or in-memory analytic tools like Qlikview, is the entire reason the data warehouse first came about. 

The decision for building or not building a datawarehouse is all about the history of the data.  Not the history that is required by law to be kept like financial data or what in many cases is considered ‘facts’ in the Kimball style.  If this were the only history needed, a data warehouse would be less necessary.  The type of history that is important is the history that cannot be reproduced within the source systems.  This is the history of changes made that are not kept by the source system.  In many cases a customer’s address may not be historically important in a transactional/source system so only  the most current record is kept.  If that history is not kept somewhere (like a data warehouse), analytics of historical purchases of products will not show a true picture of what actually happened.  It will only show the picture of what is in the source system at the current point in time.  This situation is the quinticential lynchpinn for why a data warehouse should be necessary.  The ability to track and keep history that is not kept in the source system is something SOA, or in-memory BI is not capable of reproducing.

If the desired BI capability for the business is operational in nature, a data warehouse will not offer any significant benefit over SOA.  This is a short sighted tactical means of looking at data and cannot provide strategic insight, but it certainly could be the best way to answer that need for data given the circumstances.  This would not be the end-all-be-all for BI, but it certainly can provide means to start a program.

So does this completely answer the question “Is a data warehouse necessary for BI?”  The data warehouse is necessary for a complete and sustainable BI program, but it does not have to be the start of the program.  So… of course the answer to that is still…. “It depends…”

… Doug

Since there are many types of democracies, the term Information Democracy is not easily refined. Mr. Passerini discussed his idea of Information Democracy as providing the same information at the same time to all that should view that data. This Lateral information exchange has enabled P&G unprecedented access to data propelling their decision making to be quicker and based on current data.
The purpose of their Information Democracy is to provide not only one version of the truth, but the same version of the truth to everyone. This might sound like the same concept, but there is a subtle difference and it deals with the latency of the data and ability to massage results. It tries to eliminate the “My data shows…” statements made by many because the data is owned and seen by all people at the same time. There is no delay to anyone in receiving data, no standardized reports to be re-issued, no side data to be pulled into Excel to get a different look, just the data received in a dashboard/cockpit environment.
The delivery of the data in Mr. Passerini’s Information Democracy is prolific. The same pieces of information are delivered via mobile devices, traditional PCs or P&G’s Business Sphere environment (a conference room of walls with electronic displays filled with information). The same data provided at the same time to all parties involved using multiple delivery devices allows the entire P&G managerial structure to evaluate data wherever they may be. This pervasive data culture is another example of P&Gs increased ability to adapt their business more quickly in a team environment.
The Information Democracy has not come easily at P&G as they have had to overcome obstacles. It has taken a huge effort to change the culture to embrace data for data driven solutions. Security issues make the delivering of data to all the necessary people difficult. The technology to do this is available, but the governance was generally lacking. These issues must be addressed, as P&G has, prior to successfully implementing the idea of an Information Democracy.
Transparency of the data (showing the same data to all necessary parties), timeliness of the data (getting the data to all parties as early as possible), and transportation of the data (delivering the data in multiple formats for easy consumption) make the three branches of the Information Democracy much like the executive, legislative and judicial branches make up our democracy. With these branches and the appropriate data governing processes, there truly can be an Information Democracy allowing data “…of the people, by the people and for the people.”

…Doug

Having designed and implemented Kimball style data marts and data warehouses, I was very skeptical at first.  The Kimball methodology has been proven over and over time and again.  How could something, like the data vault, that has been around a decade receive such little push from the mainstream Business Intelligence groups like TDWI?  How could this seemingly over simplified methodology really do what it claims?

As the presentation continued, Jon started asking if there have ever been issues with the Kimball style with changing the data model.    Of course there has been, hardly ever does anyone ever get the first cut at a star right.  There always seems to be things missed, either by looking in the data or by the subject matter experts forgetting to explain something.  It is one of the largest hurdles to overcome in a star design, but everyone has to deal with that… right?

One huge advantage to the data vault is a way to easily get around this issue.  If something is missed in the design, the data vault is adjusted; the star design is adjusted and reloaded.  No data is ever lost in the staging areas due to the misunderstanding of requirements because the data vault methodology works around this.  The history is kept in the vault so the ‘loading’ into the star could be re-done every time the requirements change without anything ever being lost.

Of course no data architect worth his salt would ever make a design that would lose data like that… Really? The staging areas have always been designed by the team correctly and every single star design has been flawless or the business had not changed how business is done?  The crux of this comes down to not having to ‘cook’ business logic into how the star is loaded from the source.  All the data is loaded into the data vault first and then business logic can be used to load a specific data mart/star schema.  This idea in the methodology is a genius way to never having to say ‘sorry’.

When this benefit of the data vault was covered, my initial thought was, well this is why you have a staging area.  All the data will be kept there so you can always come back to it later.  The more I thought about that idea, the more I recalled how complex the staging area becomes in trying to do this.  There is no way to model all the twists and turns needed to code this way as well as the space needed to keep a traditional ETL environment up and running.  The more I thought about it, the more I began thinking a traditional staging area and its complexities are a huge headache!  The simpler design using the data vault methodology as the persistent staging area offers huge benefits over the traditional Kimball style data warehouse staging area.  This includes repeatable code use in building and populating the data vault as well as the ability to easily account and validate the data.

Validating the data in the star can be daunting due to the business rules involved.  In order to validate the numbers found in the star, the logic used to build the star has to be applied to the source system in order to compare apples to apples.  This is not necessary in the data vault.  The simplicity of the data vault makes auditing and validation easier.  Since the data gets entered into the system as it is in the source, auditing becomes easy.

As this topic was being covered, my Kimball instincts took over and said, “Well, we have to do this anyway and can use the staging area…”  This can be true, but the simplicity of the data vault methodology makes this process so much easier and includes the ability to take commonly used data with different meanings in different departments (think customer to sales vs. marketing…) and easily link that data together.  This is no small task in the Kimball data warehouse world.

After one session and after thinking it over, I have come to the conclusion that using the data vault as the persistent staging area for a Kimball style data warehouse or the non-user accessed data warehouse for the Inmon style is the best way to allow for quick design that truly can be iterative.  The ease to get this design started and the flexibility to easily change what is presented to the user through data marts makes the data vault concept truly unique.

How many times have you told yourself . . . why didn’t I think of that? Or, how many people have gotten rich off an idea you already thought of 10 years ago?

We all are more creative than we think. The difference is in knowing how to turn a good idea into an innovation. What’s the difference? A good idea is just that – an idea. An innovation solves several problems at once and brings value to customers by helping then solve a problem they didn’t even know they had.

Innovations Aren’t Always Planned
When biologist Alexander Fleming came back from vacation and found the bacteria in one of his petri dishes had died, he didn’t view it as a failure, instead he recognized that something extraordinary had happened. From this ‘unplanned accident’, came the discovery of penicillin.

Fleming’s discovery illustrates what Yale psychologist Robert Sternberg calls ‘selective coding’. Selective Coding is the ability to distinguish important information from irrelevancies. The key is being able to detect the relevant ‘signal’ amid irrelevant ‘noise’ which is accomplished by determining critical information components that bring value to the situation. For example, is it necessary to be given instructions on how to build a watch, if all that’s needed is the time?

Possessing a unique eye to detect patterns among unlike combinations, and separating noise form news, has the capability of solving perplexing problems. This capability is what catapults an idea into an innovation.

Metaphors Make the Impossible, Possible
Turning an idea into an innovation is the ability to draw comparisons and analogies from juxtaposing elements or ideas that ordinarily don’t go together, and recognizing the hidden pattern of connections between them. It’s not enough to be able to pick out all the right or new pieces, but being ableto put them together in a new way is what is crucial.

The best tool to assist in combining unfamiliar concepts, is the metaphor . . . thinking in terms of something is ‘like’ something else. For example, who do burrs and socks have in common? Velcro. The concept of interlocking ‘hooks’ gave way to a new fastening system. A new fastener had not been invented since the mid 1800s.

Developing ideas form metaphors involves changing the way a question is worded, or brainstorming on ‘What if’ scenarios. What would happen if a coin punch and a wine press were combined? The Printing Press. What would happen if customers could order products whenever they wanted? Electronic Commerce.

Metaphors draw a mental picture. This picture is especially useful when communicating a concept that is ‘un-like’ any other product or service. Referencing something that is similar in concept, makes the unfamiliar, familiar.

Every Innovation Needs to Be Needed
Important to any innovation, is the timeliness of its purpose – a context in a relevant time. The purpose could be in response to competition, to demand or need, or in response to new regulation.

A national retail pharmacy chain responded to legislation, that requires Pharmacists, not the Technician, to dispense pharmaceuticals to customers. Without hiring additional pharmacists to support the additional tasks, the innovation came by redesigning the physical store layout and the way tasks were performed.

Once the idea of how to comply with legislation was developed, it was presented to the audience in a context that had meaning to them. Meaning for a customer may mean validating the idea against a list of criteria (relevant information) such as budget, time, and resources.

Validating the idea meant gathering information with regard to technical requirements, safety issues, production capabilities, etc. The idea at this stage of development, answered more questions than it generated. At this stage, the idea became an innovation.

In redesigning, the pharmacy layout and tasks between people, several problems were solved at once. Not only was the retail pharmacy chain able to comply with the new legislation, but as a result, an unplanned benefit occurred – prescription renewals escalated by over $200,000 per store.

The Impossible is Possible
Ken Olsen, president Digital Corporation, state in 1977 that “There is no reason for any individual to have a computer in their home.” We all learn from our mistakes and missed opportunity, however the value of innovation versus an idea, is bridging for the customer, the gap between ‘ I think it will work’ and ‘I know it will work’. It means recognizing patterns and separating necessary information from the irrelevant to create solutions that were never imaginable.

It is providing that level of detail and ingenuity that turns an ordinary idea into an extraordinary innovation.

Prepared for LUCRUM, Inc. by Susan Thomas, October 28, 2011

In 1994, Timothy O’Brien produced a play called An Object Orientation, in which two characters search in vain for a piece of lost information, only to realize that what they were really seeking was meaning, not data.

It is obvious we live in an information age where overload and information anxiety have reached epidemic proportions. What can be done to sort through what we need, and when we need it, in order to produce meaningful results?

As Thomas Stewart states in his book, Intellectual Capital, “Intelligence becomes an asset when some useful order . . .  when it is given coherent form . . . when it can be deployed to do something that could not be done if it remained scattered around . . . Intellectual capital is packaged useful knowledge.”

Classifying It
The primary purpose of intellectual capital is innovation. Human capital grows in two ways: when an organization uses the skills and experience of its people, and when more people contribute their knowledge effectively to the organization. Two principles of knowledge are classification and recognition.

Classifying and managing intellectual assets requires knowing what you are trying to accomplish. What would make a profound difference in the product or service delivery cycles? For example, the purpose of managing problems is to respond more quickly and attain a high level of customer service. The purpose of managing workload is to be able to predict requirements for staffing and adjust for peak loads. The purpose of managing of managing skills is to ensure that the right skills are in the right place at the right place and right time. The purpose of managing service levels is to monitor customer expectations.

Only when a problem has been identified then classified, can studies be conducted to isolate and resolve a particular incident. The next step is to understand the information.

Understanding It
To find order amid chaos of data overload, concentrate on providing information in context to provide meaning. It is information is context that reduces the amount of time necessary to interpret (or misinterpret) a message.

For example, it is difficult to visualize the size of an acre, if one has never seen an acre. To make the unknown familiar, provide context – an acre is about the size of two football fields (without the end zones). Announcing the pollen count is useless, unless the count is given within a range of numbers (e.g. upper and lower limits). Now I have meaningful information.

Determining what kind (classification) of information – quantitative or qualitative is important to understanding. Qualitative information is descriptive – it’s hot today. Quantitative, is digital – it is 94 degrees. Depending on your business, knowing the difference between qualitative and quantitative information and using them in the appropriate context, can at times, mean the difference between life and death.

A pilot needs to know the exact distance (quantitative) from another aircraft. It is of little value knowing the other plane is ‘very close’. In this scenario, relaying the correct classification of data (quantitative), allows the pilot to make an informed decision and correct the situation.

Information is power, but it can lead to problems. It encourages people to hoard information, manipulate it, and possibly use it as a weapon. However, an active knowledge management network tends to be self-correcting – especially if your task performance depends on it.

Managing It
Only mismanagement of customer information can explain why U.S. companies on average lose half of their customers in five years.

Knowledge sharing only works when you pay attention to organizational needs and processes of the user. Managing knowledge involves understanding the relationships between functions and people, not isolated tasks. It incorporates how well people can access, interpret and utilize the information. The goal is to allow everyone in the organization to act faster and make more informed business decisions.

A knowledge management network can be formal or informal. Informal, is manual, word-of-mouth. Formal, typically includes an electronic system relying on integrated software to make information flow. A managed knowledge network includes clearly defined business rules that standardize procedures and incorporate personal accountability.

But more importantly, a managed knowledge network lets people focus on business, not technology. The benefit to the organization is the ability to make informed business decisions quicker than the competition.

Summary
Knowledge is an advantage. It prepares your company to react to unplanned events. It allows one to identify trends in customer activity. With the speed with which organizations change, managing for competitive advantage means managing knowledge. If I understand information, I can use it. If I can’t understand it, it’s useless. Therefore, all the time spend classifying it, understanding it, and managing it is wasted.

The efficiency, the agility, with which a company can augment intellectual capital, is the true measure of its effectiveness in the Knowledge Management Age.

Prepared for LUCRUM, Inc. by Susan Thomas, October 28, 2011

Confirmed speakers for the event are:

Drew Boyd

Drew Boyd is a recognized authority, thought leader, educator, and practitioner in the fields of innovation, persuasion, and social media. He is the Executive Director of the Master of Science in Marketing Program and Assistant Professor of Marketing and Innovation at the University of Cincinnati.

Ginny Walker

Virginia (Ginny) Walker is currently GE Aviation’s Chief Enterprise Architect, reporting to the CIO. She grew up in Michigan, obtained her undergraduate degree in Systems Analysis at Miami (OH) University, an MBA from Xavier (OH), and has worked 25 years for GE in Aviation, Corporate and Energy. She has held a diverse span of Information Technology roles…from application development, to data warehouse and data management, to strategic planning, outsourcing, data center management, and ERP implementations.

Jim Goetz

Jim is the Chief Information Officer at Convergys.  He is responsible for the planning, development, and delivery of the technology products and services for the Company globally, including external and internal clients. He reports to Jeff Fox, president and chief executive officer of Convergys.

Mike Cholak

Mike Cholak is the Vice President of Customer Intelligence Services at Convergys.  He leads a team dedicated to delivering a full suite of consulting services to the Company’s clients that help leverage customer intelligence and feedback to optimize long- term customer loyalty and profitability.

For more information:

The main entities of Data Vault are Hub, Link and Satellite.

HUB Entity (HUB_): This is a defining entity. It contains a unique list of business keys. These are the keys that businesses utilize in everyday operations. For example, employee number, SSN number, Product Code. So the attributes of HUB are:

• Surrogate Key – This is a Primary Key of hub and holds 1-to-1 relationship with the Business Keys.
• Business Key – This is a Primary Key of the source system. This can be a composite key. ETL checks this key’s existence in the hub table and inserts one if it doesn’t exist.
• Load Date Time – The datetime of the key / record when it was first loaded into the table.
• Record Source – The name of the source the record originated from. This is useful for data traceability.
• Record Begin Date Time – The datetime when the record became active in the source (if available) or the datetime when ETL has been run.
• Record End Date Time – The datetime when the record is closed. This can only be detected if the logical deletes are supplied or derived in some manner.

LINK Entity (LINK_): LINKS are constructed once all the HUBS are identified. Links are relationship entities.  These are the physical representation of m-to-m 3NF relationship. It represents the relationship or transaction between hubs. The link table contains the unique list of relationships between hub keys. When a relationship arrives, it simply gets loaded into the table if doesn’t exist. Typically, the link tables translate into fact tables in the datamart access layer. For example, the link between employee number and the project number. The other attributes of LINK are:

• Surrogate Key – This is a Primary Key of the table and is useful when a link contains more than two hub keys as composite key might cause performance problems. This is also
  useful when the granularity of the link changes (a hub key is added) or history needs to be maintained on the relationships.
• Hub Key 1 to Hub Key N – The surrogate keys from the hub tables that are involved in the relationship.
• Load Date Time- The datetime when the record was loaded into the table.
• Record Source – The source system name from where the record or relationship was loaded from.

SAT Entity (SAT_): SATS holds descriptive information about the hub keys or the relationships. The satellite is most closely resembles Type 2 Dimension. When the data changes, a delta record is inserted into the table and if the certain columns changes faster than others then these can be split into two different tables to avoid data replication. For example, employee details such as employee name, address, phone number, email address in the satellite off of hub  or time spent by an employee on a certain project in satellite off of LINK that stores the relationship between an employees and projects. The other attributes of SAT are:

• Hub or Link Surrogate Key from HUB or LINK table. This is part of the primary key.
• Load Date Time – The datetime when the record was inserted into the table. This is part of the primary key.
• Surrogate Key – This is optional. It is useful when satellites have multiple values such as multiple home addresses.
• Record Source – The name of the source.
• Record Begin Date Time – The datetime when the record became active in the source (if known) or the datetime when ETL has been run.
• Record End Date Time – The datetime when the record is closed.

And stand-alone tables such as calendars, time, code and description tables may be used.

Below is a snippet of a Data Vault Model housing borrowers who have taken out Student Loans:

Modeling Rules for Each Part of the Entity:

FOR HUBS:

• Hubs keys cannot migrate into other hubs (no parent/child like HUBS).
• Hubs must be connected through links.
• More than two hubs can be connected through links.
• Surrogate keys may be used.
• Business keys are 1 to 1 relationship with surrogate keys.
• Hubs primary keys always migrate outward.
• Hub business keys and primary keys never change.
• If a hub has two or more satellites, then a point-in-time table can be built for ease of joins.
• An ‘UNKNOWN’ business key record can be inserted into Hub that can be used to tie other data in links and sats that has no business keys in source. This kind of data is usually a bad/incomplete source data.

FOR LINKS:

• Links can be connected to other links.
• Links must have atleast two hubs associated with them in order to instantiated.
• Surrogate keys may be used.
• The combination of surrogate business keys made a unique key.
• Does not contain descriptive data.
• Does not contain begin and end dates.

FOR SATS:

• Satellites may be connected to hubs or links.
• Have 1 and only 1 parent table.
• Satellites always contain either a load date-time stamp, or a numeric reference to a stand-alone load date-time sequence table.
• Primary key is a combination of ‘surrogate key’ from either hub or link and the load datetime stamp.
• Surrogate keys may not be used.
• Must have a Load End Date to indicate when the CHANGE to the data set has occurred.
• Satellites are always delta driven. Duplicate rows should not appear.
• Data is separated into satellite structures based on 1) type of information 2) rate of change.

DV model utilize bits of both 3^rd Normal Form and Dimension Modeling concepts.  This approach has made the model simple, flexible, expandable, adaptable and consistent.

• Adapted many-to-many physical relationship structure from 3NF that became a LINK table.
• The LINK table is also similar to factless fact in Start Schema.
• Adapted the notion of 1 to 1 (business key to surrogate key) tracking from dimensional modeling (type 1 dimension).
• Adapted the notion of “data over time in a separate table/structure” from dimensional modeling (type 2 dimension). This resulted in a SAT table however it is fundamentally
  different, in that it is a child dependent table, whereas the dimension is a parent table to the facts.

This is it for now. In next post(s) we will look into some examples which will show how Data Vault technique overcomes the limitations of 3NF and Dimensional Model structures when applied as an Enterprise Data Warehouse.

- Jyothi

Source: tdan.com, danlinstedt.com

iPhone4S

Like most of you, I recieve a lot of publications in my inbox each day.  This morning, an interview in Intelligent Utilitycaught my attention.  The interview, IT’s Workforce of the Future, is a very interesting observation by a CIO who has been in the workforce for less than 20 years.  In the interview the CIO, Branndon Kelley reminds us that in 1999 our workplace had better, faster technology than we had at home.  Today, for most, that simply isn’t the case.  We have a new workforce emerging that has been immersed in technology since birth and want to bring that to their new post-college positions.

Branndon issued a challenge for those of us “old timers” in IT.  Certainly our experience has taught us to be cautious and deliberate and to plan our project carefully.  These new techies, however, are more likely to jump in with both feet and more optimism.  Neither is certainly a perfect strategy but their is a lot to be learned from both.  Business is changing faster and faster every day.  When evaluating your projects here are some things you may want to consider:

1. Is there truly a compelling business reason why I couldn’t move my data to the Cloud?
2. How can I inspire my technical team to think more creatively and collaboratively?
3. Why does my staff need a permanent desk at my site? 
4. Can I find the best resources in my town/headquarters?  What if I was not limited by geography?
5. What tools do I need to get in place now so that I can enable my teams to push the boundaries of location, work hours, and technology?

Food for thought indeed.  How would you respond to the questions above?

 - Jodie