Plug into the Power of the Data Vault
April 1, 2011
In this day and age, it seems to be trendy to gravitate to the flash and splash of the latest and greatest user facing tools to address Business Intelligence issues. Some believe that if they just get a dashboard and a few nifty graphs, all of a sudden they will have “answers” flowing through their systems and into the reports. …Almost like it was magic.
The true Business Intelligence practitioners know better. Most modern systems still suffer under the same age old issues because they are still doing things the same old way. Some of the issues that are still prevalent are ability to change over time as the business changes and the integration of the information problem.
So when you build architecture on the quick and easy solution in a “silo”, you will eventually hit the wall when it comes to adaptability and scalability. So where does one turn when there is a need for speed as well as the ability to support mission critical reporting and analysis needs that must be able to pass audits?
There is a methodology that attempts to bridge the gaps between the typical issues in the current Business Intelligence offerings. The inventor of the data vault is Dan Linstedt (www.danlinstedt.com) where the concepts and rules are specified for successful engagements.
The data vault is not a product. It is not a magic pill that makes all your IT ills go away. It is a comprehensive approach to addressing real world issues with existing implementations. It brings real flexibility and adaptability to the implementation and brings reliability and dependability to the business. And with a team that understands the power of the data vault, you are now able to take your Business Intelligence environment upon which the tools that do the flash and splash can be sourced from.
According to Dan Linstedt, the inventor of the data vault methodology, the challenges around data integration include some or all of the following:
- Definition, or understanding of the data
- Functions or transformations applied to translate the data
- Interpretation of the data
- “MASTER” determination of the data
- Best storage and architecture of the data
- Visualization of the data
- Accountability and auditability of the data
- Traceability of the data
- Overloading (multi-use of single columns and record types) of data
- Historical data with lost definitions
- Data too big
- No change data capture/no audit trail
- Bad indexes
- No control over source feeds, source timing
- Multi-system valuation dependencies
- Missing data
- Changed Passwords
- Mis-aligned access rights
- Overflowing data
- Out-of-range data
- Bad domain data (a date field contains a string…)
Unless there is a comprehensive plan in place to deal with data integration, then it will only be a matter of time before your implementation will begin to suffer under the weight of the problem. Short sighted solutions only mask this issue for a short time, where our customers need a quick solution that will also stand the test of time and change.
And because of the nature of a data vault, this can be done in rapid releases that bring value within a few weeks of embarking on the project. Because of the style, there are now tools on the market that can generate the table and transformation logic. Once you reach this level, then change is transparent and accessible to your user community as well as the IT staff can finally keep close to the change as it is happening in the business.
According to Dan Linstedt, one would expect the following results from pursuing an implementation that included a data vault:
- Manage and enforce compliance to Sarbanes-Oxley, HIPPA, and BASIL II in your enterprise data warehouse (EDW)
- Spot business problems that were never visible previously
- Rapidly reduce business cycle time for implementing changes
- Merge new business units into the organization rapidly
- Rapid ROI and delivery of information to new star schemas
- Consolidate disparate data stores (i.e., master data management)
- Implement and deploy SOA, fast
- Scale to hundreds of terabytes or petabytes
- SEI CMM Level 5 compliant (repeatable, consistent, redundant architecture)
- Trace all data back to the source systems
With the data vault at the core of your Business Intelligence implementation, you are enabling your enterprise to be as nimble as possible without ignoring the core critical issues around data integration and change over time. Your user community will have the chance to grow at the pace dictated by business opportunity unconstrained by the “normal” issues around the traditional approaches.
Over the next few months, I will be going deeper into the components of a data vault, where it fits into enterprise architecture, and the ways to take advantage of the “Power of the Data Vault”. Stay tuned…
Database in the sky?
December 13, 2010
For some time now we at LUCRUM have been talking about cloud-based Business Intelligence solutions as a viable market alternative. We partnered with www.1010-data.com, offering big data – data warehouses in the cloud, and with Ocucue, offering “object” (as opposed to KPI) based data visualizations in the cloud. Executives from both of these organizations presented at our BI Symposium back in September.
Well, BI in the cloud just made a big leap forward. The poster child of cloud companies, Salesforce.com, has now entered the mix. Salesforce announced a few days ago the availability of www.database.com, a service they (salesforce) will be offering, based upon their existing architecture. WOW… The game should start moving even faster now, what with Salesforce now directly competing with the major Database vendors (Oracle and Microsoft specifically). I personally find it interesting as well that of all the areas Salesforce could have brought to the cloud (vertical/horizontal solutions, ERP/MRP systems, yet another email system, etc) they chose Database. Oh, don’t get me wrong, it makes sense relative to leveraging their existing infrastructure, but I think it’s an interesting vote on where BI, Data and the cloud are headed.
Better BI on Bigger Data
September 1, 2010
“Can we do BI without a significant infrastructure effort?”
What’s the most common BI tool? Microsoft EXCEL! If you are considering a DW initiative, you are likely encountering resistance from your hard-core, Excel based, analysts. Taking away Excel from them means that they will have less flexibility and spend more time waiting on changes. I’m guessing that they may not be on board with your vision.
1010Data is a zero-footprint, browser based data warehousing product. It has an Excel like feel, but it can handle billions of rows of data (not thousands like Excel). It’s fast, flexible and requires no infrastructure support from IT (if you are in IT, my guess is that you may have already stopped reading this article). The system is not an OLAP system, it requires no design, and not even indexing strategies – but it’s so fast!!
This is a new paradigm shift: from ETL to ELTAR – extract, load and transform as required. Hmmm….cleansing and summarization at query run time; views are used for data governance.
Don’t think it’s true? Dollar General is using the platform and was up and running in 5 weeks to 115 users with a 100% ROI.
What could you do for your customers if you could have their DW up in just a few weeks instead of a few months??
- Jodie
Data Mining – We want it Now!
September 1, 2010
“We are drowning in information but starving for knowledge.”
What do you really understand about data mining? Dr. Candace Gunnarsson was formerly a professor of statistics at Xavier University. Her view of data mining is all about getting information out of the systems you have today and predicting future results. Her view is also about needing a DW modeled in a way that makes the mining experience easier and more meaningful.
Her view is that there is prescriptive (automated) data mining and descriptive data mining. I think most people think of data mining to be an automated process. Truly, the manual exploration must happen first. What do I think the drivers should be? If it’s predicting a purchase, I need to understand you as a customer. What magazines do you read? How old are you? Male/female? How often do you come to my store? All of these become data points in an algorithm that will lead to prescriptive data mining. If I can predict that you will make a purchase, I then need to be able to test my theory and provide results back to my model. Data Mining is Avery iterative process.
To have truly effective data mining, you need to have a multi-disciplinary team. Be sure to bring in your IT, Marketing, Finance, and operations-focused team members. They will all have different views of your customers and will understand their transactional data better than anyone else. Use all of these views to create a better view.
BIG Question – can you start data mining before you have your data organized in a DW?
I’d be interested in hearing your data mining stories.
- Jodie
The Science of Visual Analytics
September 1, 2010
Today LUCRUM hosted our second BI Symposium. Once again, it was well attended and we had some great speakers! I’m hoping that this becomes a regular event. If you have yet to attend, I encourage you to come to our next event (to be scheduled).
Our first speaker was Mr. Stuart Woodward, President OD OcuCue. (http://ocucue.com/) OcuCue is an interesting start-up that’s all focused on data visualization. I always love listening to visual experts. There is such a science to visual design. It’s about understanding the psychology of how users think and perceive what they see. If you are creating a dashboard, you have to design it in the way people think – we read from left to right, heavy color should be at the lower left hand side, etc.
“Good design has two key elements. Graphical elegance is often found in the simplicity of design and complexity of data.” – Edward Tufte
Mr. Woodward’s company creates meaningful dashboards that are icon based. They go beyond speedometers and graphs and actually create a customized dashboards with icons that are meaningful to the company using them. One example that he showed was for a hospital. There are some rooms that can only take female patients or only male patients. To show bed availability, their dashboard has a pink pillow or a blue pillow to represent which rooms are available. Hmmm…never thought of that!
How are you presenting data to your users? Are you simplifying the message? Setting up the information from left to right? Are your colors meaningful? (ie Red should mean bad, green is good)
OK…gotta run and listen to the next speaker!
- Jodie
Making Information Available
August 9, 2010
I’m not sure if you’ve noticed, but I’ve not been blogging with the same gusto as of late. Ah the life of a Consultant. 
I have been working with a local financial institution creating financial models this summer. (It leaves me with little time for blogging.) I did happen to stop by our 7755 Montgomery Road office today and checked my mailbox. In it was this month’s Information Management mag. I was immediately drawn to this month’s Snapshot: Making Information Available. Here’s some stats for you to consider:
61% of respondents are less than satisfied with their current process of creating information applications and are only lukewarm about their current information application technology. Here are their complaints:
- It takes too long to assemble and deploy applications.
- It is too difficult to assemble and view information into a simple view.
- There are not enough capabilities to integreate and normalize information from disparate applications.
WOW! I ask all of you fellow BI folks out there…what are you doing to solve this problem??? Why is it with all of the tools available today, our users are finding it too difficult to use them!! What are WE doing wrong?
As I mentioned, I am working with a customer on Financial Models this summer. I am fortunate to work with some SUPER SMART people in this group. They have come up with the most ingenious ways of getting their data out of old clunky systems. They can create some of the most INSANE Excel formulas to manipulate data! Their Excel sheets are visually appealing and get data to their management in a timely manner. I’ve had some spreadsheets that have taken me days to figure out the Excel formulas (and I’m a guru!). They are awaiting IT to “build them a DW” to make their lives easier. Here’s to hoping that it can deliver on their expectations! Here’s what I would do to ensure that it does:
1. Use an iterative methodology to build the DW. Recreate existing Excel reports from the DW as you go.
2. Implement a user-friendly reporting tool that allows them to create their own reporting. Give ‘em lots of drag and drop functionality and make sure it can Export to Excel.
3. Create a request process that allows the DW to change with the Business. Creating a process that queues up the work for months and months does not help the business user to create the financial package that’s needed at the end of the month.
4. Keep the model flexible. Doing this will ensure that you can always add a new organziation, hierarchy or measurement.
5. Build cubes! These users are smart cookies and they aren’t afraid of a Pivot Table. Give them the flexibility and performance of a cube and let them start to uncover their data.
Hmmm…what’s missing from my list? What would you add?
Happy building!
- Jodie
Using Business Intelligence to Drive your own Recovery.
June 29, 2010
eWeek published a video describing the value of using Business Intelligence to find and exploit market and revenue opportunities. Great point, and very well worth the 6:49 it takes to view it. Many organizations are using BI to understand some of the basic historical results of their business. It’s the next level of organization who begins to answer questions like the below using their BI toolset:
- What are my customer’s buying is a basic question, but moreover, what products do they buy together?
- Which products do they buy when times are tough?
- What did they buy during the last recovery?
- What aren’t they buying, and what should I recommend they buy?
All great questions, and clearly a value add of a strong BI platform.
eWeek – Using-Business-Intelligence-to-Find-Your-Economic-Recovery
BI and Software-As-A-Service (BI SaaS)
May 3, 2010
Who moved my cheese…again???
Economy challenges always seem to prompt new business models and productivity increases. Remember 10 years ago and the dot.com bomb?? Prior to 1999, websites were being developed in great numbers but there was no revenue model to support it. Those companies failed…others, that found a way to take a seemingly free service and get paid for it thrived. Additionally, with the fall off in the economy, people had to find a way to deliver the same services their customers were used to but do it for less. Voila! Off-shore resources!!
In the last several years though, even off-shore resources are expensive. Seasoned IT professionals (baby boomers) are retiring and taking valuable company info along with them. Profit margins for most companies continue to erode as spending has slowed. DASD has gotten significantly less expensive and bandwidth has quadrupled (or more?)! Those “free” websites now charge fees, but they aren’t outrageous. Given these changes, it makes sense that more and more applications are moving into the Cloud.
As you know, here at LUCRUM, “we do BI”. Respoinding to our customers, we implemented Agile BI concepts long before it was fashionable. We are able to get BI projects up and running in significantly less time than our “big 6″ competitors (and do it for less!). As we continue to investigate ways to get data to our customers faster, we have become fascinated with the Cloud. Certainly there has to be a way to take all of these company assets, secure them in the Cloud and give users better/faster access to their data.
We’ve investigated a few companies that are doing this today: Good Data, OCO, BIRST, and PivotLink. What’s interesting about each of these companies is that they’ve taken the common business problems – Sales and Finance – and created models to support them. I was fortunate to participate in a meeting with Good Data last week. I’m excited to learn more about each of these companies and even more excited to see how LUCRUM can support BI in the Cloud!
Stay tuned!
- Jodie
Using Excel to run MS SQL Stored Procs
April 13, 2010
Today I had an opportunity to install a spreadsheet for a customer. It may seem simple…a spreadsheet…but the power it gave to it’s user was unimaginable.
I’ve been working in IT since 1993. In that time, I’ve become the Excel champion. Excel is cheap (relatively), installed nearly everywhere, and most everyone knows how to use it. IT rejects Excel…because it’s cheap and easy. We technicians like things to be complicated. It makes us feel smart when we can deploy an app that most people would never understand. But year after year, I find that the cheap, simple Excel spreadsheet is the one thing that can always get you a “thank you” and “you’ve made my job so easy” response from a customer.
With today’s install, I was able to get my smart guys to even look at Excel as something complex. Using a stored procedure in MS SQL 2005 and the data connection in MS Excel, we were able to create an Excel Macro that allows a user to click on a menu item and execute that stored procedure using parameters in the Excel spreadsheet.
After creating the connection, the next step was to establish parameters/prompts in the spreadsheet.
Now that the parameters were established, the user could bring back the data into a table in Excel to view the results. We chose to hide that tab and instead allow the users to manipulate with a simple but elegant Pivot Table (this did require some macro work).
THERE! DONE!
Now, with some additional time, we could clean up the macro, create some error handling and actually move the running of the stored proc to the macro…with…some…additional…time. The solution is clean and simple and the user is happy. For prototyping and user solutions that are going to just a few people, sometimes easier and faster is better.
- Jodie
Trusting Your Gut
March 29, 2010
In a recent McKinsey Article, “Strategic decisions: When can you trust your gut?”, Nobel laureate Daniel Kahneman and psychologist Gary Klein debate the power and perils of intuition for senior executives. Being in the business of business intelligence, I was at first shocked at the premise that there would be a time to trust your gut. After all, I’ve been telling execs for years that data, not intuition, should be the premise for making key business decisions. *Breathe* They agree.
What I found most interesting was that while they gave credibility to the strength of one’s intuition, they cautioned not to be seduced by overconfidence. They noted that most executives are promoted because of the confidence that they project which makes them more likely to be leaders in the first place. This confidence can cause others to trust the intuition of an exec because the exec believes they they themself are right. Hmmm…interesting point.
I think in my next BI engagement I’ll be adding a new dimension to the cube: Exec Intuition. We can plot that as a 1 (yes) or a 2 (no) and then analyze how often their intution is correct.
– Jodie