How many times have you told yourself . . . why didn’t I think of that? Or, how many people have gotten rich off an idea you already thought of 10 years ago?

We all are more creative than we think. The difference is in knowing how to turn a good idea into an innovation. What’s the difference? A good idea is just that – an idea. An innovation solves several problems at once and brings value to customers by helping then solve a problem they didn’t even know they had.

Innovations Aren’t Always Planned
When biologist Alexander Fleming came back from vacation and found the bacteria in one of his petri dishes had died, he didn’t view it as a failure, instead he recognized that something extraordinary had happened. From this ‘unplanned accident’, came the discovery of penicillin.

Fleming’s discovery illustrates what Yale psychologist Robert Sternberg calls ‘selective coding’. Selective Coding is the ability to distinguish important information from irrelevancies. The key is being able to detect the relevant ‘signal’ amid irrelevant ‘noise’ which is accomplished by determining critical information components that bring value to the situation. For example, is it necessary to be given instructions on how to build a watch, if all that’s needed is the time?

Possessing a unique eye to detect patterns among unlike combinations, and separating noise form news, has the capability of solving perplexing problems. This capability is what catapults an idea into an innovation.

Metaphors Make the Impossible, Possible
Turning an idea into an innovation is the ability to draw comparisons and analogies from juxtaposing elements or ideas that ordinarily don’t go together, and recognizing the hidden pattern of connections between them. It’s not enough to be able to pick out all the right or new pieces, but being ableto put them together in a new way is what is crucial.

The best tool to assist in combining unfamiliar concepts, is the metaphor . . . thinking in terms of something is ‘like’ something else. For example, who do burrs and socks have in common? Velcro. The concept of interlocking ‘hooks’ gave way to a new fastening system. A new fastener had not been invented since the mid 1800s.

Developing ideas form metaphors involves changing the way a question is worded, or brainstorming on ‘What if’ scenarios. What would happen if a coin punch and a wine press were combined? The Printing Press. What would happen if customers could order products whenever they wanted? Electronic Commerce.

Metaphors draw a mental picture. This picture is especially useful when communicating a concept that is ‘un-like’ any other product or service. Referencing something that is similar in concept, makes the unfamiliar, familiar.

Every Innovation Needs to Be Needed
Important to any innovation, is the timeliness of its purpose – a context in a relevant time. The purpose could be in response to competition, to demand or need, or in response to new regulation.

A national retail pharmacy chain responded to legislation, that requires Pharmacists, not the Technician, to dispense pharmaceuticals to customers. Without hiring additional pharmacists to support the additional tasks, the innovation came by redesigning the physical store layout and the way tasks were performed.

Once the idea of how to comply with legislation was developed, it was presented to the audience in a context that had meaning to them. Meaning for a customer may mean validating the idea against a list of criteria (relevant information) such as budget, time, and resources.

Validating the idea meant gathering information with regard to technical requirements, safety issues, production capabilities, etc. The idea at this stage of development, answered more questions than it generated. At this stage, the idea became an innovation.

In redesigning, the pharmacy layout and tasks between people, several problems were solved at once. Not only was the retail pharmacy chain able to comply with the new legislation, but as a result, an unplanned benefit occurred – prescription renewals escalated by over $200,000 per store.

The Impossible is Possible
Ken Olsen, president Digital Corporation, state in 1977 that “There is no reason for any individual to have a computer in their home.” We all learn from our mistakes and missed opportunity, however the value of innovation versus an idea, is bridging for the customer, the gap between ‘ I think it will work’ and ‘I know it will work’. It means recognizing patterns and separating necessary information from the irrelevant to create solutions that were never imaginable.

It is providing that level of detail and ingenuity that turns an ordinary idea into an extraordinary innovation.

Prepared for LUCRUM, Inc. by Susan Thomas, October 28, 2011

Confirmed speakers for the event are:

Drew Boyd

Drew Boyd is a recognized authority, thought leader, educator, and practitioner in the fields of innovation, persuasion, and social media. He is the Executive Director of the Master of Science in Marketing Program and Assistant Professor of Marketing and Innovation at the University of Cincinnati.

Ginny Walker

Virginia (Ginny) Walker is currently GE Aviation’s Chief Enterprise Architect, reporting to the CIO. She grew up in Michigan, obtained her undergraduate degree in Systems Analysis at Miami (OH) University, an MBA from Xavier (OH), and has worked 25 years for GE in Aviation, Corporate and Energy. She has held a diverse span of Information Technology roles…from application development, to data warehouse and data management, to strategic planning, outsourcing, data center management, and ERP implementations.

Jim Goetz

Jim is the Chief Information Officer at Convergys.  He is responsible for the planning, development, and delivery of the technology products and services for the Company globally, including external and internal clients. He reports to Jeff Fox, president and chief executive officer of Convergys.

Mike Cholak

Mike Cholak is the Vice President of Customer Intelligence Services at Convergys.  He leads a team dedicated to delivering a full suite of consulting services to the Company’s clients that help leverage customer intelligence and feedback to optimize long- term customer loyalty and profitability.

For more information:

The main entities of Data Vault are Hub, Link and Satellite.

HUB Entity (HUB_): This is a defining entity. It contains a unique list of business keys. These are the keys that businesses utilize in everyday operations. For example, employee number, SSN number, Product Code. So the attributes of HUB are:

• Surrogate Key – This is a Primary Key of hub and holds 1-to-1 relationship with the Business Keys.
• Business Key – This is a Primary Key of the source system. This can be a composite key. ETL checks this key’s existence in the hub table and inserts one if it doesn’t exist.
• Load Date Time – The datetime of the key / record when it was first loaded into the table.
• Record Source – The name of the source the record originated from. This is useful for data traceability.
• Record Begin Date Time – The datetime when the record became active in the source (if available) or the datetime when ETL has been run.
• Record End Date Time – The datetime when the record is closed. This can only be detected if the logical deletes are supplied or derived in some manner.

LINK Entity (LINK_): LINKS are constructed once all the HUBS are identified. Links are relationship entities.  These are the physical representation of m-to-m 3NF relationship. It represents the relationship or transaction between hubs. The link table contains the unique list of relationships between hub keys. When a relationship arrives, it simply gets loaded into the table if doesn’t exist. Typically, the link tables translate into fact tables in the datamart access layer. For example, the link between employee number and the project number. The other attributes of LINK are:

• Surrogate Key – This is a Primary Key of the table and is useful when a link contains more than two hub keys as composite key might cause performance problems. This is also
  useful when the granularity of the link changes (a hub key is added) or history needs to be maintained on the relationships.
• Hub Key 1 to Hub Key N – The surrogate keys from the hub tables that are involved in the relationship.
• Load Date Time- The datetime when the record was loaded into the table.
• Record Source – The source system name from where the record or relationship was loaded from.

SAT Entity (SAT_): SATS holds descriptive information about the hub keys or the relationships. The satellite is most closely resembles Type 2 Dimension. When the data changes, a delta record is inserted into the table and if the certain columns changes faster than others then these can be split into two different tables to avoid data replication. For example, employee details such as employee name, address, phone number, email address in the satellite off of hub  or time spent by an employee on a certain project in satellite off of LINK that stores the relationship between an employees and projects. The other attributes of SAT are:

• Hub or Link Surrogate Key from HUB or LINK table. This is part of the primary key.
• Load Date Time – The datetime when the record was inserted into the table. This is part of the primary key.
• Surrogate Key – This is optional. It is useful when satellites have multiple values such as multiple home addresses.
• Record Source – The name of the source.
• Record Begin Date Time – The datetime when the record became active in the source (if known) or the datetime when ETL has been run.
• Record End Date Time – The datetime when the record is closed.

And stand-alone tables such as calendars, time, code and description tables may be used.

Below is a snippet of a Data Vault Model housing borrowers who have taken out Student Loans:

Modeling Rules for Each Part of the Entity:

FOR HUBS:

• Hubs keys cannot migrate into other hubs (no parent/child like HUBS).
• Hubs must be connected through links.
• More than two hubs can be connected through links.
• Surrogate keys may be used.
• Business keys are 1 to 1 relationship with surrogate keys.
• Hubs primary keys always migrate outward.
• Hub business keys and primary keys never change.
• If a hub has two or more satellites, then a point-in-time table can be built for ease of joins.
• An ‘UNKNOWN’ business key record can be inserted into Hub that can be used to tie other data in links and sats that has no business keys in source. This kind of data is usually a bad/incomplete source data.

FOR LINKS:

• Links can be connected to other links.
• Links must have atleast two hubs associated with them in order to instantiated.
• Surrogate keys may be used.
• The combination of surrogate business keys made a unique key.
• Does not contain descriptive data.
• Does not contain begin and end dates.

FOR SATS:

• Satellites may be connected to hubs or links.
• Have 1 and only 1 parent table.
• Satellites always contain either a load date-time stamp, or a numeric reference to a stand-alone load date-time sequence table.
• Primary key is a combination of ‘surrogate key’ from either hub or link and the load datetime stamp.
• Surrogate keys may not be used.
• Must have a Load End Date to indicate when the CHANGE to the data set has occurred.
• Satellites are always delta driven. Duplicate rows should not appear.
• Data is separated into satellite structures based on 1) type of information 2) rate of change.

DV model utilize bits of both 3^rd Normal Form and Dimension Modeling concepts.  This approach has made the model simple, flexible, expandable, adaptable and consistent.

• Adapted many-to-many physical relationship structure from 3NF that became a LINK table.
• The LINK table is also similar to factless fact in Start Schema.
• Adapted the notion of 1 to 1 (business key to surrogate key) tracking from dimensional modeling (type 1 dimension).
• Adapted the notion of “data over time in a separate table/structure” from dimensional modeling (type 2 dimension). This resulted in a SAT table however it is fundamentally
  different, in that it is a child dependent table, whereas the dimension is a parent table to the facts.

This is it for now. In next post(s) we will look into some examples which will show how Data Vault technique overcomes the limitations of 3NF and Dimensional Model structures when applied as an Enterprise Data Warehouse.

- Jyothi

Source: tdan.com, danlinstedt.com

The first entry is the slide deck from the recent Advanced Architecture Conference in Denver: http://danlinstedt.com/datavaultcat/datavault-advanced-architecture-conference-slides/

There is a wealth of information contained herein that outlines at a high level the benefits and reasons why the data vault method is enabling pervasive BI. It goes into details about comparing 3rd normal form and the star schema form to the value the data vault brings to the table. It outlines the components of a data vault, including the hubs, links, satellites, PIT tables, and bridge tables.

It was nice to see the connection with the methodology to emerging tools. There are now more than a handful of tools that are undertaking the task of automating the ETL and loading processes of an EDW. The method of the data vault allows for a lot of the technical innovation. Dan presents the case well that businesses today want answers faster and cheaper today as well as having the system flexible enough to evolve with the business over time. Most EDW systems today fall short of being pervasive, and often that is a methodology problem that appears to have an answer.

In addition to these topics, Dan pushed forth the proposition of an “Operational Data Vault” and began to lay out how this connects to Business Intelligence. This section was very interesting and helped to turn on a few light bulbs and progressive thoughts. I love the idea of pushing the intelligence as far as possible while still keeping the style and benefits. I agree with Dan that this is still so new that there are not yet any vendor applications or tools that can help with this…yet.

The presentation does leave a little bit of gap with the more advanced concepts, such as change management and how to leverage a data vault using agile principles. It is also a bit light on how to drive a project plan as well as what to watch out for while attempting to build a data vault style EDW. All in all, he presents a very compelling case as to why the data vault should be strongly considered when delivering Business Intelligence.

The second entry is a public rebuttal to a challenge about the validity of the data vault methodology:
http://danlinstedt.com/datavaultcat/datavault-new-response-to-frank-habers-part-1/

As Dan states, he felt the need to lay out the facts and correct misconceptions from the original author. After you get past the first few sections of disclaimers and background, the article does a great job of comparing and contrasting methodologies and technical styles for building the EDW.

I often see HDSA (persistent historical data storage area) implemented at client sites and they sometimes go by different names (ODS, staging, etc.). I have yet to see a better discussion of the merits of a data vault compared to the HDSA as he presents here. If you are in an environment that has an ODS or persistent history area, please take a moment to read this section, because the warnings and issues are real and one should not make decisions here lightly because of the long term ramifications.

Dan also goes into great detail to itemize the benefits and value proposition of why you take the steps to have a data vault at the core of the EDW.

I would say that the overall tone of the article still makes it a bit hard to read, but if you can overlook the defensive stance, you will find many gems that will help solidify and justify the purpose and value of the data vault compared to any other technique.

BI systems need DATA and every business has terabytes of real data which can provide them with the information and knowledge they need to make the right decisions on time. But the key is to turn that data into information in a timely, efficient and effective manner once the WHAT AND WHY questions are answered i.e., what information is needed, what matters and why that is required.  In today’s market, every business is in a RACE. The race to conquer others. The race to generate more gains/profits. The race to foresee the risks early on so that they can be avoided.  So time is of the essence here.

An optimized BI system integrates large volume of external and internal near real time data to allow management to create opportunities by making intelligent decisions after performing predictive analysis of their approach on the business. A good BI System is like a GPS. An effective GPS is one that not only shows you a route to your destination but also guides you when you hit roadblock, gives up-to-date external conditions (constructions / traffic) information, provides multiple routes to choose from, suggests you with alternatives for shorter and fastest routes, predict the total time based on your driving behavior, tells you what to expect next etc. Just knowing the path to your destination is not sufficient. You need to know many other factors during the whole ride to reach destination on time and without any hurdles.

For a good integrated BI system, a good Data warehouse architecture needs to be in place.  Data warehouse architecture is “an integrated set of products that enable the extraction and transformation of operational data to be loaded into a database for end-user analysis and reporting”. Below are the pictorial representations of different “flavors” of DW architectures.

Methodologies used by different architecture:

Kimball’s DW Architecture – Is based on ‘Bottom-UP’ methodology.

Inmon’s DW Architecture – Is based on ‘Top-Down’ methodology.

Dan Lindstedt’s Data Vault DW Architecture – Is based on ‘HYBRID DESIGN’

The first two design methods have some limitations for Data Warehouse layer such as inflexibility and unresponsiveness to the changing departmental needs during the implementation phase, insufficient auditability of data back to its source system, inability to integrate unstructured data, inability to rapidly respond to changes (organizational changes, new ERP implementations) or difficult to load type 2 dimensions in real time. This is where DATA VAULT came in to rescue. Data Vault follows a ‘HYBRID DESIGN’ methodology which follows ‘TOP-DOWN ARCHITECTURE WITH A BOTTOM-UP DESIGN’.

The model is a mix of normalized modeling components with type 2 dimensional properties. In this model, the DW serves as a backend system that houses historical data which is integrated by the business keys. All data ‘good, bad, incomplete’ gets loaded into the data vault and all the cleansing and application of business rules takes place downstream i.e., out of DW. This means that Data Vault model is geared to be strictly a data warehouse layer, not as a data delivery layer which still requires physical or Virtual star schemas or cubes for Business Users or BI tools to access.

Bill Inmon in 2008 stated that the “Data Vault is the optimal approach for modeling the EDW in the DW2.0 framework.”

In Part 2 and 3, I am going to explain different components of Data Vault and it’s power with the help of some examples.  That will clearly explains why the Data Vault should be a preferred “flavor” for different businesses.

- Jyothi Kaparthi

The way to approach testing a Business Intelligence (BI) system is to get the business to have ownership and buy-in early and often. The business users should be writing test cases and be responsible for executing them from a business perspective as well as trains them on the content in the system. The technical people should be ready to assist with query development or whatever to help the testing to be completed.

There should be some validation that is part of the design of the Extract, Transform, and Load (ETL) process itself. Some of this is to make sure mechanically things happened as they should and that there are appropriate logs when they don’t. In addition, the ETL developers should perform some kind of UFI (Unit Function) testing prior to moving to a TEST environment as well as a code review or peer review. Depending on the complexity of the ETL process, one generally don’t test each component of the process due to the details involved, but focus more on the net result of the test (i.e., all rows were inserted with no errors and all columns contain values – what happened in between is not as important to test because the load was successful).

In addition, the technician should take the next step of developing quality controls that make sure what was in the final table structures is what was expected. For example, have a report from the Operational Data Store (ODS) area that groups and sums some business keys with some key metrics and compared them to the results from the new implementation area, while highlighting only variances. This should be sent to a data governance team every morning. As long as it was clean, the BI team is sure mechanically things were working pretty well.

Depending on some of the business rules implemented, one may need to have reports that highlight “Unknown” values and other things that need to be dealt with by the business. Some of these scenarios should become test cases. The business should be using the Ad-hoc environment to be writing reports and queries to test the results. Ultimately, these reports should be reviewed by the data stewards as part of the data governance initiative.

For the most thorough results as well as the highest quality BI environment, everywhere there was a business rule implemented, there should be a test case that verifies the rule was implemented correctly. Depending on the volume and complexity, one may need to prioritize them and tackle the most important ones first.

For the documentation, it can be as simple as keeping a spreadsheet with the following items:
• Test Case #
• Test Case Description
• Tester
• Date Tested
• Expected Outcome
• Actual Outcome
• Pass/Fail

It is critical, as originally stated, to get the business users involved in the testing of the deliverables. There have been cases where the business thought what they were using to compare balances with was correct, but were eventually convinced the BI application was correct and they had a broken business process instead. This is most difficult because it is on a case by case basis, but this usually becomes the biggest challenge and hurdle that needs to be overcome to be perceived as successful. Ultimately the business must provide you with the information to know whether “the values put into the Data Warehouse or BI dashboard are correct”. And you are completely dependent on the business rules they gave you are correct (a lot of times they aren’t in version 1)…and even more risky if there is no data governance process in place.

A word of caution, if you don’t get the business buy-in on testing, they will certainly blame you when things aren’t correct in production (especially if that happens for things that were overlooked in testing). It is wise to have a step where the business had to sign off on testing and that they were comfortable with what was moving into production and that was very helpful when issues arose. Because they were involved in the process and it was not mostly IT doing the testing, finger pointing was kept to a minimum. In addition, shared success and teamwork was fostered bridging the gap between business users and Information Technology (IT) groups that sadly exists in a lot of organizations.

Therefore, if you want to jump start your BI implementation, I suggest considering the following steps to help you minimize rework and avoid common pitfalls that lead to higher costs and longer delivery time:

1. Change the name of the dimension TIME_DMN to DATE_DMN because there should be a date and a separate time dimension available in the EDW. It is not unusual to see this combined together, but there is much more flexibility with a separate physical date dimension and time dimension as well as much fewer rows.

2. Use a tool to load all the date value possibilities desired instead of coding this by hand. Perform a one time “seeding” of the dimension and be finished with it. (See http://www.kimballgroup.com/html/MDWT08Content/Ch07_Date_Dim_2000-2020.xlsx). Why reinvent the wheel when the experts have spelled it out for all to use?

3. Develop as much of the ETL code as close to the database as possible. The database will give you the best performance if you can get away with it. I generally use ETL tools to call a SQL statement (views) in the database and write into another object in the same database. The ETL tool is more of a job control mechanism rather than a comprehensive logic flowing mechanism. This does require a methodology that keeps the ETL pattern based and simple like a data-vault.

4. Use a “data vault” instead of the ODS as the core of the data warehouse to store all history. Loading should be from files to a staging area. Load from the staging to data-vault and from data-vault to data-mart. The ODS could be used as an online repository for the files or loads and should closely resemble the file layouts and no business logic applied. The ODS may serve other purposes as well and may need to be treated as another source system.

5. Use a tool or combination of tools to generate ETL code and database structures as well as document. These tools exist on the market today and obviously code generators will consistently outperform even the best coders (see http://thefuturevalueofbusiness.com/data-vault-applying-the-story-of-john-henry.htm).

6. Fact tables should all be numeric columns. In a perfect world and for performance reasons as well as logic, stick to numbers. Try to move all non-numeric columns to dimensions. The databases will thank you when the table has lots of rows but the width of the rows is as narrow as possible.

7. Constraints in a star schema should be “RELY” – the ETL process should guarantee the validity. This one relates to Oracle, but is useful to make a point that the downstream systems will still see the constraints but the database does not need to incur the overhead of maintaining the relationship. The ETL process will do a fine job of making sure this is correct.

8. Check constraints are unnecessary as well (check for NOT NULL) on tables. This is more overhead that can be avoided. This is more important in transaction systems and again the ETL process should be dealing with this if it is important.

9. Dimensions should have indexes on columns that are often used in GROUP BY clauses. Stars will perform better if there are indexes against the columns that are mostly used to group and filter on.

10. The attributes about the ETL job should be stored in a separate table with a FK in all the tables that point back to the run. This helps keep fact tables lean and aids in consistency of the ETL process itself. It may also help in the cleanup effort of errant ETL runs.

11. Data governance should be happening now to review, cleanse, establish business rules, and sign off on the master dimensional data. It’s never too early to start on this process. The sooner lists and value sets are presented to the business around key business elements the earlier the discussion can begin to resolve issues and assign ownership.

12. Information that is used for reporting should never be destroyed (Preliminary vs. Final) unless signed off by the business. Once the information is loaded into the BI environment, it is a wise idea to keep it around. At some time the question will be asked “Why did we make THAT decision?” Keeping this information off-line or on-line is a different discussion.

13. All character columns should be defined as VARCHAR(255) instead of variable smaller widths in order to handle most column size situations. I see this often and I am puzzled as to why it is still being done this way. Modern databases are pretty good at storing variable length fields and not wasting space in the process. You will have a problem once you come across a new data source that has a value that is wider than what you sized the existing column. This sometimes comes along after a new acquisition gives you a new data source for your business.

14. Development should not be locked down. It stifles productivity and creates a bottleneck. There still needs to be peer and code reviews but that should happen prior to promotion to the Test environment while things are being designed and developed. Source control helps here too. Prototyping requires quick response and the faster things can change while the design is being “vetted” the better.

15. Use database sequences or identity columns instead of using what the ETL tool provides. It is generally a bad idea to rely on the ETL tool to set the “surrogate” keys. Moving code between environments becomes a real challenge. Databases have built functionality for this purpose, so it’s a good idea to use it.

Even implementing one of these steps will be beneficial, you should see marked improvement if you can do the whole list.

I am sure there are many more steps. I would love to hear your feedback and experience. Have you tried the steps and had a different outcome? What steps would you add or remove?

Like other “Big Men” such as Paul Bunyan and Pecos Bill, John Henry also served as a mythical representation of a group within the melting pot of the 19th-century working class. He grows to become the greatest “steel-driver” in the mid-century push to erect the railroads across the mountains to the West. When the owner of the railroad buys a steam-powered hammer to do the work of his mostly black driving crew, to save his job and the jobs of his men, John Henry challenges the owner to a contest: himself alone versus the steam hammer. John Henry beats the machine, but exhausted, collapses and dies.

His story is usually seen as an archetypal illustration of the futility of fighting the technological progress that was evident in the 19th century upset of traditional physical labor roles. Some labor advocates interpret the legend as illustrating that even the most skilled workers of time-honored practices are marginalized when companies are more interested in efficiency and production. Although John Henry proved himself more powerful than the steam-drill, he worked himself to death and was replaced by the machine anyway.

Note: the above excerpt was taken from http://en.wikipedia.org/wiki/John_Henry_(folklore).

So how does this story apply to the Information Technology industry in the 21st century? I find in my travels a similar resistance to efficiency and production, although it would be due to different motives. In this day and age, the most opportune ways to drive down costs over the long term is to automate as much as possible. I still see and talk to technicians that are still trying to build and manage the entire Business Intelligence environment by hand, basically wearing themselves out trying to keep up with maintenance and the rate of change demanded by business requirements.

So what is the root cause of the resistance? I would suggest that there is a comfort level in the technologist in us that entire works hard to reach a level of competence and skills and then, once attaining that level, resists enabling technologies. This is akin to John Henry feeling that he was able or better than a machine in doing a particular job. We, myself included, are comfortable with what we know how to do. Fear holds us back as well, because if we are able to generate the ETL job process, we will be replacing ETL developers with machines. Change requires us to leave our comfort zone and see beyond our own fears, in order to deliver real value to our customers.

In the information technology industry, as it has evolved over time, components move more to a modular or commodity approach as efficiency and refinement processes are applied over time. You see this with software products on the market today that have “wizards” that do routine or repetitive tasks. The more repetitive tasks can be commoditized, the more efficient the design and development process becomes. This saves real time and real dollars on development projects.

So what is one component, among many, to target for this approach? So in the realm of Business Intelligence, there is a pattern based approach to data modeling called the Data Vault ( www.learndatavault.com ). This pattern is manifested in three basic types of structures: hubs, links, and satellites. The nature of these structures allow for a repetitive process to be involved in the loading of the Enterprise Data Warehouse (EDW). For the most part, information is inserted, not updated or deleted. It tracks all changes over time, is integrated, and is auditable back to the source system. From the EDW, the standard data marts can be generated (and in some cases, this can be done with views). Because the pattern is restricted to three basic types, it becomes simpler to generate the loading ETL process using a machine. It is still not a perfect science, and depends on the value of the source model – but the point is the more that can be generated will save time and money over the long term.

So how does the technologist address the “John Henry” in each of us in order to provide the best service to our customers? First, one has to admit that it is good and beneficial to have a machine generate code and processes that were once done by hand. Secondly, one has to be open-minded to realize the benefit. Thirdly, one has to have the courage to take on change, by realizing that inside the change component, there is opportunity for future growth. This is manifested in customers being pleased that high quality projects are produced within a predictable cost and timeframe. By products of this method also reduce support costs over time. It’s a win for the technologist as well as success for the customer.

It’s time to embrace the “machine”, and allow it to do what it does best. The technician needs to move on ahead and add the thought leadership to the projects that will harness the power of smart design with efficient development patterns. The knowledge and knowhow is here today. I know. I have seen it done. I will continue to lay out the design patterns over the next few entries of what it takes to move in this direction and leverage the power of the data vault to bring world class solutions to our clients.

IDAPT Solutions COMPLEX PROJECTS

Well, BI in the cloud just made a big leap forward.  The poster child of cloud companies, Salesforce.com, has now entered the mix.  Salesforce announced a few days ago the availability of www.database.com, a service they (salesforce) will be offering, based upon their existing architecture.  WOW…  The game should start moving even faster now, what with Salesforce now directly competing with the major Database vendors (Oracle and Microsoft specifically).  I personally find it interesting as well that of all the areas Salesforce could have brought to the cloud (vertical/horizontal solutions, ERP/MRP systems, yet another email system, etc) they chose Database.   Oh, don’t get me wrong, it makes sense relative to leveraging their existing infrastructure, but I think it’s an interesting vote on where BI, Data and the cloud are headed.