Since there are many types of democracies, the term Information Democracy is not easily refined. Mr. Passerini discussed his idea of Information Democracy as providing the same information at the same time to all that should view that data. This Lateral information exchange has enabled P&G unprecedented access to data propelling their decision making to be quicker and based on current data.
The purpose of their Information Democracy is to provide not only one version of the truth, but the same version of the truth to everyone. This might sound like the same concept, but there is a subtle difference and it deals with the latency of the data and ability to massage results. It tries to eliminate the “My data shows…” statements made by many because the data is owned and seen by all people at the same time. There is no delay to anyone in receiving data, no standardized reports to be re-issued, no side data to be pulled into Excel to get a different look, just the data received in a dashboard/cockpit environment.
The delivery of the data in Mr. Passerini’s Information Democracy is prolific. The same pieces of information are delivered via mobile devices, traditional PCs or P&G’s Business Sphere environment (a conference room of walls with electronic displays filled with information). The same data provided at the same time to all parties involved using multiple delivery devices allows the entire P&G managerial structure to evaluate data wherever they may be. This pervasive data culture is another example of P&Gs increased ability to adapt their business more quickly in a team environment.
The Information Democracy has not come easily at P&G as they have had to overcome obstacles. It has taken a huge effort to change the culture to embrace data for data driven solutions. Security issues make the delivering of data to all the necessary people difficult. The technology to do this is available, but the governance was generally lacking. These issues must be addressed, as P&G has, prior to successfully implementing the idea of an Information Democracy.
Transparency of the data (showing the same data to all necessary parties), timeliness of the data (getting the data to all parties as early as possible), and transportation of the data (delivering the data in multiple formats for easy consumption) make the three branches of the Information Democracy much like the executive, legislative and judicial branches make up our democracy. With these branches and the appropriate data governing processes, there truly can be an Information Democracy allowing data “…of the people, by the people and for the people.”

…Doug

The main entities of Data Vault are Hub, Link and Satellite.

HUB Entity (HUB_): This is a defining entity. It contains a unique list of business keys. These are the keys that businesses utilize in everyday operations. For example, employee number, SSN number, Product Code. So the attributes of HUB are:

• Surrogate Key – This is a Primary Key of hub and holds 1-to-1 relationship with the Business Keys.
• Business Key – This is a Primary Key of the source system. This can be a composite key. ETL checks this key’s existence in the hub table and inserts one if it doesn’t exist.
• Load Date Time – The datetime of the key / record when it was first loaded into the table.
• Record Source – The name of the source the record originated from. This is useful for data traceability.
• Record Begin Date Time – The datetime when the record became active in the source (if available) or the datetime when ETL has been run.
• Record End Date Time – The datetime when the record is closed. This can only be detected if the logical deletes are supplied or derived in some manner.

LINK Entity (LINK_): LINKS are constructed once all the HUBS are identified. Links are relationship entities.  These are the physical representation of m-to-m 3NF relationship. It represents the relationship or transaction between hubs. The link table contains the unique list of relationships between hub keys. When a relationship arrives, it simply gets loaded into the table if doesn’t exist. Typically, the link tables translate into fact tables in the datamart access layer. For example, the link between employee number and the project number. The other attributes of LINK are:

• Surrogate Key – This is a Primary Key of the table and is useful when a link contains more than two hub keys as composite key might cause performance problems. This is also
  useful when the granularity of the link changes (a hub key is added) or history needs to be maintained on the relationships.
• Hub Key 1 to Hub Key N – The surrogate keys from the hub tables that are involved in the relationship.
• Load Date Time- The datetime when the record was loaded into the table.
• Record Source – The source system name from where the record or relationship was loaded from.

SAT Entity (SAT_): SATS holds descriptive information about the hub keys or the relationships. The satellite is most closely resembles Type 2 Dimension. When the data changes, a delta record is inserted into the table and if the certain columns changes faster than others then these can be split into two different tables to avoid data replication. For example, employee details such as employee name, address, phone number, email address in the satellite off of hub  or time spent by an employee on a certain project in satellite off of LINK that stores the relationship between an employees and projects. The other attributes of SAT are:

• Hub or Link Surrogate Key from HUB or LINK table. This is part of the primary key.
• Load Date Time – The datetime when the record was inserted into the table. This is part of the primary key.
• Surrogate Key – This is optional. It is useful when satellites have multiple values such as multiple home addresses.
• Record Source – The name of the source.
• Record Begin Date Time – The datetime when the record became active in the source (if known) or the datetime when ETL has been run.
• Record End Date Time – The datetime when the record is closed.

And stand-alone tables such as calendars, time, code and description tables may be used.

Below is a snippet of a Data Vault Model housing borrowers who have taken out Student Loans:

Modeling Rules for Each Part of the Entity:

FOR HUBS:

• Hubs keys cannot migrate into other hubs (no parent/child like HUBS).
• Hubs must be connected through links.
• More than two hubs can be connected through links.
• Surrogate keys may be used.
• Business keys are 1 to 1 relationship with surrogate keys.
• Hubs primary keys always migrate outward.
• Hub business keys and primary keys never change.
• If a hub has two or more satellites, then a point-in-time table can be built for ease of joins.
• An ‘UNKNOWN’ business key record can be inserted into Hub that can be used to tie other data in links and sats that has no business keys in source. This kind of data is usually a bad/incomplete source data.

FOR LINKS:

• Links can be connected to other links.
• Links must have atleast two hubs associated with them in order to instantiated.
• Surrogate keys may be used.
• The combination of surrogate business keys made a unique key.
• Does not contain descriptive data.
• Does not contain begin and end dates.

FOR SATS:

• Satellites may be connected to hubs or links.
• Have 1 and only 1 parent table.
• Satellites always contain either a load date-time stamp, or a numeric reference to a stand-alone load date-time sequence table.
• Primary key is a combination of ‘surrogate key’ from either hub or link and the load datetime stamp.
• Surrogate keys may not be used.
• Must have a Load End Date to indicate when the CHANGE to the data set has occurred.
• Satellites are always delta driven. Duplicate rows should not appear.
• Data is separated into satellite structures based on 1) type of information 2) rate of change.

DV model utilize bits of both 3^rd Normal Form and Dimension Modeling concepts.  This approach has made the model simple, flexible, expandable, adaptable and consistent.

• Adapted many-to-many physical relationship structure from 3NF that became a LINK table.
• The LINK table is also similar to factless fact in Start Schema.
• Adapted the notion of 1 to 1 (business key to surrogate key) tracking from dimensional modeling (type 1 dimension).
• Adapted the notion of “data over time in a separate table/structure” from dimensional modeling (type 2 dimension). This resulted in a SAT table however it is fundamentally
  different, in that it is a child dependent table, whereas the dimension is a parent table to the facts.

This is it for now. In next post(s) we will look into some examples which will show how Data Vault technique overcomes the limitations of 3NF and Dimensional Model structures when applied as an Enterprise Data Warehouse.

- Jyothi

Source: tdan.com, danlinstedt.com

BI systems need DATA and every business has terabytes of real data which can provide them with the information and knowledge they need to make the right decisions on time. But the key is to turn that data into information in a timely, efficient and effective manner once the WHAT AND WHY questions are answered i.e., what information is needed, what matters and why that is required.  In today’s market, every business is in a RACE. The race to conquer others. The race to generate more gains/profits. The race to foresee the risks early on so that they can be avoided.  So time is of the essence here.

An optimized BI system integrates large volume of external and internal near real time data to allow management to create opportunities by making intelligent decisions after performing predictive analysis of their approach on the business. A good BI System is like a GPS. An effective GPS is one that not only shows you a route to your destination but also guides you when you hit roadblock, gives up-to-date external conditions (constructions / traffic) information, provides multiple routes to choose from, suggests you with alternatives for shorter and fastest routes, predict the total time based on your driving behavior, tells you what to expect next etc. Just knowing the path to your destination is not sufficient. You need to know many other factors during the whole ride to reach destination on time and without any hurdles.

For a good integrated BI system, a good Data warehouse architecture needs to be in place.  Data warehouse architecture is “an integrated set of products that enable the extraction and transformation of operational data to be loaded into a database for end-user analysis and reporting”. Below are the pictorial representations of different “flavors” of DW architectures.

Methodologies used by different architecture:

Kimball’s DW Architecture – Is based on ‘Bottom-UP’ methodology.

Inmon’s DW Architecture – Is based on ‘Top-Down’ methodology.

Dan Lindstedt’s Data Vault DW Architecture – Is based on ‘HYBRID DESIGN’

The first two design methods have some limitations for Data Warehouse layer such as inflexibility and unresponsiveness to the changing departmental needs during the implementation phase, insufficient auditability of data back to its source system, inability to integrate unstructured data, inability to rapidly respond to changes (organizational changes, new ERP implementations) or difficult to load type 2 dimensions in real time. This is where DATA VAULT came in to rescue. Data Vault follows a ‘HYBRID DESIGN’ methodology which follows ‘TOP-DOWN ARCHITECTURE WITH A BOTTOM-UP DESIGN’.

The model is a mix of normalized modeling components with type 2 dimensional properties. In this model, the DW serves as a backend system that houses historical data which is integrated by the business keys. All data ‘good, bad, incomplete’ gets loaded into the data vault and all the cleansing and application of business rules takes place downstream i.e., out of DW. This means that Data Vault model is geared to be strictly a data warehouse layer, not as a data delivery layer which still requires physical or Virtual star schemas or cubes for Business Users or BI tools to access.

Bill Inmon in 2008 stated that the “Data Vault is the optimal approach for modeling the EDW in the DW2.0 framework.”

In Part 2 and 3, I am going to explain different components of Data Vault and it’s power with the help of some examples.  That will clearly explains why the Data Vault should be a preferred “flavor” for different businesses.

- Jyothi Kaparthi

The true Business Intelligence practitioners know better.  Most modern systems still suffer under the same age old issues because they are still doing things the same old way.  Some of the issues that are still prevalent are ability to change over time as the business changes and the integration of the information problem.

So when you build architecture on the quick and easy solution in a “silo”, you will eventually hit the wall when it comes to adaptability and scalability.  So where does one turn when there is a need for speed as well as the ability to support mission critical reporting and analysis needs that must be able to pass audits?

There is a methodology that attempts to bridge the gaps between the typical issues in the current Business Intelligence offerings.  The inventor of the data vault is Dan Linstedt (www.danlinstedt.com) where the concepts and rules are specified for successful engagements.

The data vault is not a product.  It is not a magic pill that makes all your IT ills go away.  It is a comprehensive approach to addressing real world issues with existing implementations.  It brings real flexibility and adaptability to the implementation and brings reliability and dependability to the business.  And with a team that understands the power of the data vault, you are now able to take your Business Intelligence environment upon which the tools that do the flash and splash can be sourced from. 

According to Dan Linstedt, the inventor of the data vault methodology, the challenges around data integration include some or all of the following:

• Definition, or understanding of the data
• Functions or transformations applied to translate the data
• Interpretation of the data
• “MASTER” determination of the data
• Best storage and architecture of the data
• Visualization of the data
• Accountability and auditability of the data
• Traceability of the data
• Overloading (multi-use of single columns and record types) of data
• Historical data with lost definitions
• Data too big
• No change data capture/no audit trail
• Bad indexes
• No control over source feeds, source timing
• Multi-system valuation dependencies
• Missing data
• Changed Passwords
• Mis-aligned access rights
• Overflowing data
• Out-of-range data
• Bad domain data (a date field contains a string…)

Unless there is a comprehensive plan in place to deal with data integration, then it will only be a matter of time before your implementation will begin to suffer under the weight of the problem.  Short sighted solutions only mask this issue for a short time, where our customers need a quick solution that will also stand the test of time and change.

And because of the nature of a data vault, this can be done in rapid releases that bring value within a few weeks of embarking on the project.  Because of the style, there are now tools on the market that can generate the table and transformation logic.  Once you reach this level, then change is transparent and accessible to your user community as well as the IT staff can finally keep close to the change as it is happening in the business.

According to Dan Linstedt, one would expect the following results from pursuing an implementation that included a data vault:

• Manage and enforce compliance to Sarbanes-Oxley, HIPPA, and BASIL II in your enterprise data warehouse (EDW)
• Spot business problems that were never visible previously
• Rapidly reduce business cycle time for implementing changes
• Merge new business units into the organization rapidly
• Rapid ROI and delivery of information to new star schemas
• Consolidate disparate data stores (i.e., master data management)
• Implement and deploy SOA, fast
• Scale to hundreds of terabytes or petabytes
• SEI CMM Level 5 compliant (repeatable, consistent, redundant architecture)
• Trace all data back to the source systems

With the data vault at the core of your Business Intelligence implementation, you are enabling your enterprise to be as nimble as possible without ignoring the core critical issues around data integration and change over time.  Your user community will have the chance to grow at the pace dictated by business opportunity unconstrained by the “normal” issues around the traditional approaches.

Over the next few months, I will be going deeper into the components of a data vault, where it fits into enterprise architecture, and the ways to take advantage of the “Power of the Data Vault”.  Stay tuned…

Well, BI in the cloud just made a big leap forward.  The poster child of cloud companies, Salesforce.com, has now entered the mix.  Salesforce announced a few days ago the availability of www.database.com, a service they (salesforce) will be offering, based upon their existing architecture.  WOW…  The game should start moving even faster now, what with Salesforce now directly competing with the major Database vendors (Oracle and Microsoft specifically).  I personally find it interesting as well that of all the areas Salesforce could have brought to the cloud (vertical/horizontal solutions, ERP/MRP systems, yet another email system, etc) they chose Database.   Oh, don’t get me wrong, it makes sense relative to leveraging their existing infrastructure, but I think it’s an interesting vote on where BI, Data and the cloud are headed.

• What are my customer’s buying is a basic question, but moreover, what products do they buy together?
• Which products do they buy when times are tough?
• What did they buy during the last recovery?
• What aren’t they buying, and what should I recommend they buy?

All great questions, and clearly a value add of a strong BI platform.

eWeek – Using-Business-Intelligence-to-Find-Your-Economic-Recovery

The first and probably most important non-technical differentiated aspect of iStream is the concept of Alignment.  Many consulting organizations and internal IT organizations have some type of design or planning step often called “Envisioning” as an initial step in their development process.  This is for good reason:  understanding the customer’s end goal or picture of success is critical to the success of the project.  At the same time, this does not procedurally support the fact that many individuals are involved in determining the success of a project, and further, in most cases these individuals are not in detailed agreement in regards to what that success looks like, or how it is defined.  Alignment takes this into account, and is a prescribed process to ensure a common understanding of the success criteria by the key stakeholders involved in any enterprise project, including department heads and/or the Information Technology department.  This includes a focus on ensuring that a miscommunication cannot occur where language is not specific enough, for example in clarifying the accepted definition of the term “Sales” in a company.  To explore this a bit, is “Sales” the number of transactions? The dollar volume closed?  Over what timeframe? By what channel? (sales people, resellers, distributors, telesales, etc.) As simple as this concept may sound – misunderstandings or assumptions in areas as simple as this are generally a key reason for project failure.  In this area, LÛCRUM is unique and differentiated in its development approach.

Another key differentiation of LÛCRUM’s approach, particularly as it relates to Business Intelligence, is in the concept of the iteration of a project.  The iStream methodology allows for iteration in the development of the end result, particularly through the recognition that many pieces may make up the whole.  For example, related to Business Intelligence; we may begin by working with an individual decision maker, say the VP of Marketing.  In working with this person we may offer to them the YourView Instant Analytics solution, allowing them to rapidly see their information in a new way through the combination of several different reports or sources into a single view.  Per the YourView solution, this can take place in a matter of days; however by definition it follows the iStream process – however abbreviated – as it is focused on only a single user.  When that VP is prepared to create a complete solution for the Marketing department, the initial work now functions as a pilot/proof of concept rolling into the Alignment, Discover and Architect components of iStream for the larger YourView 360 (Data Mart) project.  In this fashion, we are “iterating” our development of the data mart through one or more “Instant Analytics” projects.  Both projects follow iStream; however the smaller engagements feed into the larger.  When that organization is prepared to roll out an Enterprise Data Warehouse – the same holds true, the work that had been completed at the Data Mart level for the Marketing department will now be employed in the Alignment, Discover and Architect phases of the Enterprise Data Warehouse project.  In this fashion the work that we accomplish at any level of the Business Intelligence Solution chain is applicable for the next, and all would be accomplished using iStream.

While the items above are not descriptive of the entirety of iStream, nor of the entire list of benefits of the LÛCRUM approach, they are absolutely two of the components of iStream which differentiate it from the plethora of SDLC approaches available in the market, and another aspect of what makes LÛCRUM a unique Business Intelligence Consultancy.

To get at the data that most executives feel they need to make accurate decisions, many turn to the manual modification of existing reports, or the creation of their own “Pet” spreadsheet they use almost daily, or certainly many times a week. 

 In an update to a report cited last spring on this site, a September, 2009 Dartmouth University study suggests that the error rates in formulas on spreadsheets in their study were only .087% of all formulas they audited.  HOWEVER, these were in cases where the formula produced the WRONG RESULT, and actually resulted in 87% OF THE SPREADSHEETS REVIEWED having errors in which the spreadsheet then produced the wrong result. 

How good is good enough?  What if you could reproduce the “Pet” spreadsheet in a true Business Intelligence solution which would ensure that the data and results in the sheet were as solid as the data in your transactional systems in the first place?  How much does the wrong data or the wrong decision cost you, or your company?  I would argue that “good enough” might just be good enough, if you could ensure that the data was accurate, and mitigated the possibility of error, while increasing the timeliness of the information to the decision maker.  We have deployed such systems in a couple weeks’ time leveraging tools like SharePoint, Excel, and other software products that our customers already owned, and quickly delivered a system to our customer where we dramatically increased the accuracy of their information.  These solutions form the basis of our iterative approach to Business Intelligence.

So sure, I could find what I was looking for…”what’s the temp at 9AM?”  It takes a pair of readers a few seconds, but it’s there.  There are so many other data points, however.  Is it getting warmer or colder?  Is it going to rain today?  Certainly the other data points are there that would help me to make the decision – relative humidity, cloud cover, wind speed – but I may need to consult the company metadata to understand what it all means and if those numbers mean it will be getting hotter or colder.  THIS IS DATA.  Your org has it…you’ve got to make sense of it. 

What LÛCRUM does, is make this DATA meaningful.  We like to call it Business Intelligence or Data Visualization.  Simply stated, we take all of those data points and help you to make better business decisions (or in this case, help you to decide if you should wear a coat or bring your umbrella). 

THIS IS BI!

THIS 

IS 

BUSINESS 

INTELLIGENCE!! 

Taking lots of data and making it meaningful…yeah, that’ s what LÛCRUM does. 

- Jodie

iPad

I wrote a post earlier today prior to seeing the official “unveiling” of the iPad.  After watching 10 minutes of the broadcast, I trashed that version.  Here is what I saw in the iPad…

1.  It’s a iPhone on steroids.  All of the things I love about my iPhone are here.

2.  It’s cool…yeah, I mean it…cool. 

3.  It replaces the newspaper…with a newspaper.  You wanna read the NY Times?  They’ll have an app for that.  With a crystal clear 9″ x 7″ screen it will be better than the newspaper.  I sure hope my Cincinnati Enquirer gets on board with this!  Here’s a few more specs for you:  http://www.pcworld.com/article/187870/ipad_specs_what_apple_announced_and_what_we_still_dont_know.html

4.  For business apps, like mail, I can type longer messages with the larger keyboard.  My fingers won’t get tired from the one key at a time keystrokes.  This will be great when out of the office.  I’ve often waited to get back to the office to respond to a mail message that I say when I was out, simply because my response would take to long on the little keyboard.  Even blog posts could be written on the iPad.

5.  Small & lightweight – THIS is the device to take with you on your next trip out of town!

So it has no hard drive…so what!  With Google docs, I don’t need one.  If I keep my files on SharePoint, I can use Lucrum’s own iPhone app – Attache to access those docs.  (Hey Steve – the SDK is already available for the iPad…can you work on that Attache app??)

Wow!  So excited!  Wonder how long I’ll be able to keep from buying one of these??

- Jodie