Applying Big Data Philosophy To Silos

Can we use big data architecture to solve big company problems?

One of the most significant challenges to overcome in any organization is silos - those technological, political and budget-driven islands of activity.  Email is here; ecommerce is there; social is in another department; and display is managed outside.  Each produces data that might be useful in achieving objectives, but they are locked up.

We know, or at least believe, that we need to deliver a seamless experience based on a 360-degree view of the consumer.  What we do not know a priori is what tactical campaigns or even strategies might be appropriate.  Since insights change how we market, rather than just tweak a plan, we need to explore and test first.  And that's the rub - I can't define a complete data architecture because I don't yet understand how I'll use it.

The typical, big company approach is to kick of a project lead by IT to design an integrated solution based on structured data models and ETL (extract, transform and load) processes.  The vision is a common company database that can be shared and leveraged appropriately through-out the company.  I've worked on those projects throughout my career.  But is that the right approach in light of the realities of today's needs?

When data management sources were scarce, ie expensive, then it paid dividends to design the solution first. In the early data warehouse days, we'd estimate building a marketing solution at nine months and $1,000 per gigabyte.  Today, I'm not so sure we have that much time due to the volume, velocity and variety of data. 

Because we're dealing with a data fire hose, we can't design first because the sources will change before the ink dries on the umpteenth version of the Visio diagram.  In fact the whole notion of applying structure to unstructured data is somewhat of an oxymoron.  We need to get the data together first, start working with it and finally apply some structure.  Since Big Data architectures put the data design at the end of the process, maybe they're appropriate for more pressing tasks - like looking at the integration of silos.

The design stage needs to shift to focusing on the environment rather than the contents.  The key marketing questions actually become technology ones: How will we stream data into a useful place?  How can we reduce the time to get answers to questions?  How will we marry on and offline data? How do we deploy insights? 

So, I think it is time to jump in and see what I can learn from email and display data.