Democratizing Data
June 25, 2009
Wired magazine has a great interview this month with America’s first ever CIO, Vivek Kundra, who has been tasked with making the vast amounts of data collected by the Federal Government available for public use. Kundra talks candidly about the need to open up this information to the people, and the power that can come from analyzing and understanding data. The project is now coming to life on the site data.gov. Sources for data include the EPA, Library of Congress, FBI, National Science Foundation…. and on and on and on, with reports ranging from peanut recall data to most wanted lists to on time reports for airlines.
Here is an excerpt from the article explaining the goal of the site:
"The goal of Kundra’s new Web site, Data.gov, is create a place where all the information is easy to find, sort, download, and manipulate. He wants to put as much data out there as possible, then sit back and let the private sector come up with great ways to use it. He envisions a future in which well-designed spreadsheets, charts, and graphs are embedded in applications for phones, Facebook, and blogs."
This quote speaks to the power of data in our world. Certainly our government has more of this raw material than anyone, and opening it up to be refined and tapped into by citizens and businesses will help to create new breakthroughs in our world. Data gives us the ability to better understand our world. Of course it often must be refined, shaped, and combined with other pieces of data to become useful information. Once information is created, we have the opportunity to see our world in new and exciting ways. it becomes the basis for informed debate, enlightened creativity, and compelling innovation.
By opening this data up to the public, the collective wisdom of the nation and even the world is being enhanced. People everywhere will have greater insight, deeper understanding, and ultimately a better definition of the truth. What to do then is a whole different debate, yet one that can also be guided by data.
The data.gov site is by no means perfect. It is definitely still a work in progress. There are broken links, some reports and files have limited formats, it is clunky and cumbersome, there are limited feeds, and there is not yet much data from individual states. Still, it appears that the site will continue to add enhancements, data sources, and useful functionality to address these issues. Even in its current imperfect state, data.gov has the potential to deliver great information.
Something else I learned from the article is that Kundra is embracing concepts like cloud computing, software as a service and open source development – placing the government further up the innovation curve than I would have guessed. It is plesantly surprising to see such things.
Mr. Kundra closes the article with a quote that I really like – "By democratizing data, the American people will be able to hold their government accountable, based on evidence rather than talk." Politics has no shortage of talk on both sides of the aisle. It is great to see that perhaps data will play a bigger role in governing our country, informing our citizens, and advancing our economy. While I would never wish for data to replace talking, I am hopeful that it provide us with more intelligent things to say.
And Data for All: Why Obama’s Geeky New CIO Wants to Put All Gov’t Info Online
Shocking Statistics on Spreadsheets
June 3, 2009
A number of recent studies have shown that, among other things, up to 94% of spreadsheets used today contain errors. Read more
Moderation
May 14, 2009
Yesterday, I was fortunate enough to have been chosen to moderate a panel discussion put on Technology First in Dayton, Ohio. The event was entitled “My Web 2.0 Story,” and featured a diverse and talented panel made up of the following professionals:
- Alan See – Berry Network
- Doug Ross – Western Southern
- Tony Blankemeyer – PicsMatch
- Joshua Smith – NCR
- Neil Arthur – Dayton Business Journal
The panel was great – Alan is a Marketer, Doug is an IT Guru, Tony a young entrepreneur, Joshua is an HR professional, and Neil a newspaper publisher. They provided an excellent mix of viewpoints, backgrounds, and skills. Each panel member contributed unique ideas to the conversation, and they worked surprisingly well with one another to articulate their views. I dropped in my two cents from time to time as well.
Common themes included transparency, service, and innovation. There was considerable discussion around the idea of using web 2.0 as a business tool for listening – both internally to the needs of employees and externally to customers, competitors, and influencers. Finally, just about everyone recommended that the best way to understand web 2.0 was, as Alan See put it “to get in the water and swim.”
Interestingly there was considerable discussion around the measurement of social media. I am fascinated by this concept, and view this as an emerging area of opportunity in the technology space. The ability to measure not just activity on the social web, but to then link it to internal metrics and measures is fascinating. I am currently exploring this space as part of my job here at LUCRUM, so it was good to hear so many questions about it. (I will be discussing this more in upcoming posts, but suffice it to say I am excited about it.)
Andy Hickey, with the Technology First, had the great idea of having the tables construct questions during the lunch portion of the meeting at the begining. I then called on the tables throughout the meeting to ask questions – sort of analog web 2.0 style. We bounced back and forth between my questions and those of the crowd – all of which were exceptional.
I made a number of new connections at the meeting and picked up some great ideas as well. My thanks to Andy, Ann, the panel, the crowd, and everyone who made the event so much fun.
If you were there, I would love to hear your thoughts on the event. What did you like? What didn’t you like? What was the best question / answer you heard? Who was your favorite panelist? Chicken or beef? Please share your comments…
Handy Advice
April 2, 2009
If you spend a considerable amount of time working on a computer every day, you may have experienced carpal tunnel syndrome to some degree. I found myself wearing a wrist brace a few years back while battling this painful condition. If you or someone you know is experiencing pain, you may want to check out this short video from percussionist David Kuckhermann in which he shares some simple exercises he uses to avoid the pain.
If that is not enough, check out these tips from software engineer Dan Hersam. The cubicle can be a dangerous place. Stretch, use proper equipment, and go easy on the Mountain Dew. If these stretches are the most exercise you get in a given day, at least you will still be able to text. Thanks to Lifehacker for the tips.
Stories
December 18, 2008
What happens after we roll our applications into production? We are left with the stories?
It’s like raising a child, isn’t it? We spent so much time and pour our lives into the development of those much needed applications. As we trade hours for dollars as we watch them grow…
A problem turns into an opportunity.
That opportunity turns into a solution.
That solution gets a sponsor and a team.
The team raises that young solution.
Suddenly it blossoms into a application…a production application!
It may have taken us weeks, months or even years, but we were given guardianship over that child for a period of time. As it grew, and that production day came closer and closer, we felt the awkward anxiety of letting go. Yes, we know the dangers out there waiting for our young application; potentially low chance of adoption, push back, rejection, even the fear of death.
That young application is safest in the test environment. At least we can control its exposure and limit its risks. But in our heart of hearts, we know that the application was build for production. We know that it was designed to take on the hardest of situations and out perform expectations – that this application will make an impact in the business; making things better, faster, cheaper!
That day comes and we let it go and watch it fly. For a period of time, we watch it closely and ensure its safety. We anticipate the missteps and try to head them off. We add to it and extend its capabilities as new challenges are seen. We pour more and more of our time and energy into that application.
Soon, it grows mature and reaches the stage of life where it takes on its own personality. It reaches critical mass and there is no slowing it down. That application forms new relationships and makes new friends.
Then in a cruel twist of fate, it forgets about you. After all, you have other children to raise. While we may look back in pride, our jobs demand we look forward with a combination of power, agility and leveraged experiences to make an impact for the future.
Now the question remains; “What are we to do about this?” The answer is easy but it does draw against your heart. The answer, my friend, is that we are left with the stories, those fascinating stores of how we did it. Yes, some people call that experience, but who wants to hear about experience when they can listen to a story.
What are your favorite stories?
I’ll never forget leading a large government Oracle-based BI project back in 1998 when you couldn’t get an Oracle person for 100 lbs of gold. I brought in 4 network engineers – yeah you guessed it, they were Novel network managers. I trained one to be a DBA and the other three to be developers. What a time that was; gathering requirements, navigating the murky waters of government contracting, designing complex integration between the US Air Force and the US Navy, data architecture and applications development…and at the same time training four people in the art of Oracle! The point that I remember most was bringing that last person on board. You see, the contract stated that the person had to have a 4 year degree. Well, that last person had only an A.S. degree – well to be exact, she had two A.S. degrees. I remember it well; walking into the General’s office to get an exception. It had come down to that…asking a General! I walked into his office and after I gave my 2 minute context introduction, I said “Sir, this person has two 2-year degrees and the last time I checked…2 plus 2 equaled 4″. Then I shut my mouth and waited for what seemed to be a very long time and he looked me up and down and finally gave me the go ahead to move forward.
What attachment do I have with that application that is still in production today? My attachment is through those stories which make up the sum of my experiences. It is from these stories that I am forming who I am and amassing my worth.
Happy Story Telling!
~Scott Felten
It’s 11 O’Clock. Do You Know Where Your Databases Are?
November 11, 2008
Where are your databases? (And what is your strategy for knowing…)
Saying and doing are two different things. This is true and it’s also a good delineation along the lines of databases. How’s that you say? Well, for one thing, I have never worked anywhere recently (last dozen years or so) that had one database technology in play. Different products have different value propositions, especially over time – as new features and capabilities are rolled into existing product lines and even new database technologies are introduced. Recently I had an engagement where they used IBM’s DB2, Oracle 10g, Sql Server, Sybase, My Sql, Informix and even Red Brick (never heard of that one before). Sounds like it’s out of control, doesn’t it? Well, to tell you the truth, it wasn’t and here’s how I got there.
First of all, they had a strategy. This strategy was built on a common taxonomy of terms that was agreed upon by all parts of the organization. These terms were used to classify and underpin the database technologies. With these terms, there was a foundation and a single version of understanding in which to base conversations and negotiations on. Here is an example that might be helpful for you.
Proposed (or Candidate) – This category represents those technologies (databases) that someone or a team would like to be considered as part of the enterprise architecture. The process has not yet started for technologies with this classification, but they have been identified as having some initial momentum towards inclusion.
New (or Emerging) – This term does not reflect the current state of the technology, but rather the current state in the context of the organization. For example, the technologies with this classification have been approved and accepted as part of the enterprise’s strategy, but with a limited focus. These technologies indicate that there is a high degree of oversight in when, how and where they are used. It could well be that there is a need for DB2 (which is not a new technology by any stretch of the imagination), and in this case it could be that DB2 is ‘emerging’ in the context that the organization does not yet recognize this as a standard, but has agreed that it can be (should be) used within limited scope with oversight.
Standard (or Approved) – This term denotes that these technologies and products are used within the normal business practices. When these products are leveraged, there is not much discussion on scope and the oversight is usually limited to the technical framework in place versus the bigger discussions of should we and could we and how would it impact the rest of the organization and so on… We like to have standards so that we can focus our attention on solving business problems and not on political positioning and other side issues.
Contained (or Restricted)- This term refers to those technologies that are ‘restricted’. These are non standard and are usually relegated to existing applications. This is an important part of the strategy. While the scope is known, the oversight is focused on limiting the proliferation of these technologies for reasons of enterprise impact on the overarching strategy.
Declining (or Phasing Out)- This term refers to all those technologies that are being identified as ones we wish to sunset. Here this technology, while still in production, is not allowed for any new use and a plan is in place (that also includes a sunset date) for its removal from the organization.
Retired (or Removed) – This term refers to those technologies that are not allowed to be used at all under any circumstances within the organization. A formal write up of the justification and historic retirement plan must accompany the technology.
One important thing to note is that this taxonomy of classification is not linear. It could be that My Sql was identified as a candidate and after the proper feasibility discussions and general impact analysis, it was promoted to emerging. After some experience and discussions, it might be that My Sql becomes established within a niche use. When this is the case, My Sql would be promoted to contained; not that we want to phase out the product, rather, we want to liberate is growth but highly restrict it to a certain scope of use.
Start to understand your environment and where your technologies are located within the database strategy 101. Set goals to identify those that are too costly and create a strategy to sunset them. At the same time, look out on the horizon and see those new (or new to you) technologies that may play a part in your organization in the (near) future – start to embrace them and set processes up to manage them as they flow from term to term within the organization. Determine what your real standards are and socialize them. It could be that your headaches of the future can be averted if you properly set up a containment strategy. The benefits are there for you to harvest, but you have to do your part by leading the way.
Once we know what we have and classify those database technologies (or any technology or practice), we can then focus on the implementation of the strategy; Better, Faster and Cheaper. For without this mature taxonomy and disciplined approach, we will struggle to get a hold of our infrastructure and we need to be able to not only get a hold of our infrastructure, but to wrestle it to the ground an show it who is boss!
Enjoy the Journey!
~ Scott Felten
When You Have Trust
October 28, 2008
The other day we had a technical team meeting where we were discussing access rights for a global solution that we have begun to implement but have a need to extend some functionality. Well, it gets complicated pretty quickly. To be honest, I started drawing out all the potential combinations of variables that can occur and I’m trying to lead the group through this thought.
Well, this one “rogue” guy (I think to myself) keeps bringing up an idea. I hear it but it doesn’t seem to make sense to me. So, I discount it and try to get the group to move on. He keeps bringing it up and won’t let it go. We all tell him that he sometimes has listening problems and to hang it up for a bit.
Does he do this? No. Not because he is really a “rogue” guy or because he enjoys making things difficult (I have encountered my share of these people), rather, James knows that he can trust the group. James feels the trust and sees it played out over time. James knows that the environment is mature enough to handle the truth. Remember that line, “You can’t handle the truth” from the movie A Few Good Men. There was no trust in that relationship – that is for sure!
James sticks to his guns and won’t let it go. He did so in an encouraging way, but forceful. Finally, it dawned on us that he was right. Had James not felt the trust, he would avoid the conflict by either shutting down and ceasing to contribute or becoming defensive; and we would not have the proper solution to the problem. Way to go James (you know who you are)!
By the way, I called him later that day and said thanks and that I appreciated the way he handled himself. I thanked him for putting the needs of the group ahead of the need for him to avoid unpleasant conflict. This was a highlight for both of us that day. We saved a few days time and money and walked away with a better solution for less money.
In a recent blog post I wrote “This is a key point (gaining trust) because without trust the team is guarded and people don’t share. A solid foundation of trust is necessary for any team that wants to be highly successful.” Conflict is a necessary part of a team. But healthy conflict doesn’t focus on people, rather it focuses on the topic at hand.
Did you ever experience a team that had conflict and someone when historical? Not hysterical, historical! You know; “…yeah, but remember when you did this and you said that and he said… and she said…” This is because there was a lack of trust. When this happens, people focus on defense and of course sometimes a good defense is a good offense. Other times they shutdown and withdraw. Either way, that conflict is not healthy.
Conflict + Trust is healthy debate and leads to innovation!
Conflict w/o Trust leads to murder (of at least one’s character).
Go find someone to Trust.
~ Scott
Not Just Another Object-Relational Mapping Service
October 16, 2008
Andy Erickson and I went to a Microsoft MSDN Unleaded event last week (10.07.2008) and they uncovered the new ADO.NET Entity Framework. It’s a new tool to help developers bridge different data sources into a single representation in code. Stated a different way, a customer entity can exist in two different systems but be represented in code as a single customer object. It abstracts the logical schema in the database and its conceptual schema for the application. That, in it self is powerful; but there’s more. The ADO.NET Entity Framework provides services to generate all the business objects and database access for you on the fly. You can sort, page, and filter data using LINQ with very little code. Consume the service with an ASP.NET application or couple it with a Windows Communication Foundation service for some real bang in the enterprise!
To learn more, check out the MSDN article: http://msdn.microsoft.com/en-us/library/aa697427(VS.80).aspx
You can also download the sample code from the MSDN presentation: http://blogs.msdn.com/wsteele/attachment/8970691.ashx You will need the Adventure Works database installed and SP1 for the .NET 3.5 framework in order to build and run the sample application.
Please post any questions and comments in the comments section.
Happy coding!
Business Intelligence, Done Right
October 14, 2008
Business Intelligence efforts often fail, not because of the technicians, but because of the disjointed relationships we (IT) have with the business. We fear relationships. We think that our investments and communications are a waste of time because we have a tremendous amount of downward pressure to deliver – so we start quickly and build according to some document that we substitute as face time with our partner. To fix this problem, we need to take the best part of our building experience as ITers and put that together with the best part of relationship management – skills that we often don’t posses, but are vital to successfully delivering value and meeting our partners expectations.
{more on “The Right Way” below}
Kimbal’s methodology calls for the segregation of the three layers of BI and this is fantastic. There are in fact three separate BI tracks that can and should run concurrently. The three tracks are:
1. Technical Architecture
2. Data Architecture
3. Application Architecture
After all, we need to make progress on the technical choices of architecture and tools. What is our environment like, what standard tools do we use, what is our technical strategy to frame and contain our BI effort?
At the same time, we need to start looking at our data. What data will we need to work with, what is it, how should it relate, where does it come from, how do we extract, transform and load it. These are considerations of the data architecture effort.
Again, at the same time, what level of innovation do we need to arrive at? How do we present this to the user and provide the desired interaction to help mine data to craft information.
Kimbal’s Methodology
This methodology is right on! It is triggered by and the scope is managed by the requirements. These requirements are derived from that box that is labeled “Program/Project Planning”. But it’s not enough, it’s not there yet! In my mind, this is a bit limited and really only addresses the “Go” portion of the project – to build. All too often, BI projects fail because of thing that are ‘above’ or before the Go. Here is what I have found to be the differentiating factors.
Requirements should be built from Strategy. Why are we building this BI application? What are the important questions that need to be answered? What is the value that these answers bring; are we making things better or faster or cheaper? A well formulated strategy addresses these questions and should manage the deliverables. When we rely on a set of requirements to build a BI solution, we are abstracted from the strategy and this is not good. When this abstraction occurs, we are removed from the business value and are relegated as a commodity…simply build to this specification.
However, strategy is not born without a birthing process. The birthing process consists of a couple of elements. The first is an alignment or a focus at most senior levels. If we can’t align on the true aspects of the business, we are forced to work in a vacuum and can, at best produce a solution that is severely slanted. This slant often caters to the loudest voice or the biggest ego. Worse, our solution is then forever raised in a silo that contributes to expensive growth and maturity with a limited enterprise appeal. If and only if we get this alignment, we are doomed to deliver value slices rather than holistic value.
This alignment is not enough. After we have a focus, we then have to dive deep within the organization to see what raw materials we have to work with. I refer to this phase as ‘Discovery’. Once we are aligned and focused on the business value, what do we have to work with? What data exists in what format and at what availability and quality?
To state this concisely, we have the following necessary phases that will lead us to success.
1. Alignment – A common focus with enterprise level buy in.
2. Discovery – An exploration to ascertain our resources and an evaluation to ensure it supports the common focus.
3. Strategy – The actual strategy that is an outcome of “What do you want” + “What do we have”.
4. Requirements – The plan of how to get there from here.
Now, we are ready to “Go” forth and multiply! We are ready to build! And we know what we are building (requirements) and how the effort will benefit the organization (strategy). We know what we really have to work with (discovery). And finally, we have those committed at the top who are committed at the deepest levels of the effort and will both support/defend the effort as well as ensure that its accepted and reused across the enterprise.
Putting it all together, let me suggest the following methodology…
Sometimes we are asked “Why are you just sitting there? Do something!” We need to rally and fight the urge to just do something. We need to change that around as ask “Why are you just doing something? Sit there” (and get focused, aligned, discover, produce strategy, birth requirements…then we can “Go” and do something excellent).
The difference between a commodity and a partner is the level of time you invest in the relationship.
Take the time to do it right and it will be celebrated!
The next series of blogs will outline the above components; alignment, discovery, strategy, requirements and the three team approach to building BI solutions. Feel free to shoot me some questions or thoughts you may have regarding these topics.
Enjoy the ride!
~ Scott Felten
The Data Governance Color Palette
September 24, 2008
It is no secret that people are different. My wife loves process and order. She is disciplined and pays attention to detail. I, on the other hand, love chaos and new things. I like to create and invent.
Then there are those accountant/analyst folks who love details and facts. Finally, there are those who are people people – the ones who pay careful attention to others and their feelings. True, these are simply attributes that we all have in varying degrees. But, it is no secret that we have some in big doses while others are very small.
So, what happens when governance groups are out of balance? What if we have a data governance group that is made up of ten people like me…loving new things and big visions with little to no attention to detail? Or, there are ten folks like my wife – loving processes and steps to implement. Not a whole lot of new out of the box thinking – but truly skilled in the art of process.
Also, imagine ten folks sitting on the data governance group that are those people people – they think of relationships and peoples feelings. Are these people going to be ‘enforcers’ of the rules and standards? Finally, how about those factual lovers? Can they implement?
We need to have a well balanced governance team that has the right amount of visionary ability plus the people needed to validate those ideas against facts and figures plus those important people people who can tell you how to craft that message and how to reach people plus those implementers, those who can frame the plan in all its glory!
Well, if you have been around governance groups at all, you have seen cases where the team is not well-rounded. There are just too many of the same type of people in the group – or one ‘attribute’ exists in a dominate individual which infects the entire team.
Consider the following:
Where the yellow are the folks who are visionaries, always thinking up new things, but don’t have much skill at details, working with people or process.
Where the blue are the folks who are based in facts and figures.
Where the red are the folks who are those people people.
Where the green are the folks who are highly skilled in the art of process.
I was thinking about these different attributes and what happens when we have an imbalanced group. What are the warning signs, what can we expect when we are unbalanced. In the above diagram, I list the three roles of governance at the top and the different attributes in the color bands. Crossing them are the things that seem to happen when that group is out of balance.
For example, I have seen this many times…a team that is unbalanced and mostly blue (based in facts and figures) spends lots of time to produce the “10 Commandments”, then they post them on the world for all to see. These are the folks who develop principles and standards with complete definitions and taxonomies. These become artifacts or works of art to hang up and be proud. But nothing happens after this? Why, because they needed the greenies!
Or take the unbalanced team in the yellow zone… lots of big ideas come from this team. We often see a tremendous amount of momentum out of the gate because they sold senior leadership who moved mountains to take advantage of these ‘game changing’ ideas. But, very soon their progress stops, more than stops – we simply never hear from those folks again. Why, because those ideas were not tested by the blue folks!
Another example is an unbalanced green team… they spend tons of time to put together standard operating procedures which stream line operations. But this group misses the mark because they didn’t have that spark that a yellow brings!
Finally, when those red folks who are so needed in the group are in the majority – we have to be careful that we don’t approach the governance from the touchy feeling perspective, where we may be more interested who’s feet we may step on.
I have only touched on four cells – there are twelve to consider. Think it through. We need all these attributes and need to have them in a balanced manner. Look for those warning signs and think about who is in your group. Do you need to bring about change? Start by knowing where you are, looking at where you have been, so that you can make those adjustments to take you where you want to go!
Happy Balancing!
~Scott Felten
Polls
Loading ...