Breaking Data Silos - Three Keys For Success

Posted by Eliran Boraks on Jan 31, 2019 4:06:00 PM

Three Keys For Successfully Breaking Data Silos to Unlock New Business Insights

Data languishes in silos in every enterprise. Most enterprises also sponsor at least one initiative to free that data so every part of the organization can mine it for insights. The challenge is that these “silo-breaking” projects are complex, expensive, require organizational change, have a long time-to-value period, and do not scale. Many “silo-breaking” initiatives actually result in more siloed data environments because each initiative applies a different approach. No wonder so many of them fail.

At Cambridge Semantics, we engage in many silo-breaking projects because our Anzo platform’s capabilities make them relatively easy, quick and repeatable. But this post isn't about Anzo. It's about the three key lessons learned over the years from those projects:

  1. To mine value from data, it must be easy to explore.
  2. Data governance and lifecycle are critical. Users require confidence that data is current and accurate.
  3. Influencing organizational change with Minimum Viable Products (MVP) is effective, if not essential. Organizational change is difficult, expensive and requires executive buy-in, so use Proofs of Concept (POC) and MVPs to prove value.

To mine value from data, it must be easy to explore.

Making data easy to explore sounds obvious, but it's easier said than done. For example, data often comes from different systems with different structures, naming conventions and many more variations.

The solution is to transform the source data structures into a business-friendly data model (schema). An easy-to-understand model is key to making it easy for users to explore data and find insights.

But a business-friendly model introduces a new problem: It can't include all possible fields and structure from the contributing source systems. As a result, the model limits the types of analysis possible. The solution is to maintain links between the new business-friendly model and the source data models. This approach enables users to traverse all the datasets. As users’ experience with the system increases, they dig deeper.

Data governance and lifecycle: users require confidence that the data is current and accurate.

When applications connect data from multiple sources on a recurring basis, eventually they miss a beat. The system must cope with anomalies. To expect the unexpected, the system requires the following basic functionalities:

  • Provide transparency with business users and auditors about data freshness, data collection, and data transformation.
  • Track data lineage and display how the system collected, aggregated and transformed the data.
  • Provide real-time reports and audit logs that show when data arrived, when the system processed it, where the data is, and other details.
  • Send automated alerts to stakeholders when something goes wrong. Don't wait for end users to tell you "We’re missing data." Be proactive.

Data lineage diagram shows users how data flows in the system.

Workload Status

Workload status and data freshness status are critical for helping users feel confident about the system.

Influence organizational change with proofs of concept and minimum viable products.

As I mentioned in the beginning, silo-breaking projects are complex, time-consuming and expensive. They also require the organization to adapt to change, which creates challenges related to policies and culture.

The best way to minimize this risk is to build an MVP that demonstrates the potential in a low-risk environment. MVPs require discipline to ensure they stay within budget and schedule. For more information about MVPs, I recommend reviewing the Wikipedia page on MVP.

Silo-breaking projects must be repeatable to scale. Enterprises must employ standards and best practices to avoid creating new silos. Standard technologies and methodologies enable organizations replicate the process and ultimately connect all enterprise data.

Summary: unlock the value of siloed data.

Breaking silos and harmonizing organizational data is necessary to find new insights and unlock that data’s full potential. To ensure your project is successful, make sure it’s easy for users to explore the data to find new insights, provide transparent data quality to users, prove the concept with MVPs and ensure the process is repeatable.

Tags: Data Management, Data Integration, Data Governance, Anzo, Insight Platforms

Subscribe to the Smart Data Blog!

Comment on this Blogpost!