Once upon a time, around 2005, in a land called… let’s say Denver, a mysterious collector arrived knocking at office doors asking for data and promising in return all the riches of the world. The office keepers listened in rapture as the collector told great feats of optimization. Show us! they cried, show us how you streamline and accelerate! Show us how you revolutionize and transform!
Okay, the collector said, bring me your dreams and I shall make them come true. But I’m going to need your data engineering team. Clear their calendars. Also, I only do the buttons in emails. If you want to do other stuff, you need to get other collectors. Unfortunately, none of us are on speaking terms. Soon the offices of Denver were filled with the chatter of collectors ignoring each other.
You get it. In the last decade, companies collected vast reserves of data, because collection technology existed. All the business teams used the data they collected in their own workflows in their own enablement tools. “Siloes.” Companies have lots of data, but most of it isn’t being processed, connected, or activated. The economic opportunity of AI — McKinsey predicts trillions of dollars added to the global economy — is in extracting the value of all the data companies have already collected. It’s exciting.
Data Orchestration: An Operational Definition
The process by which companies extract value from data is data orchestration. In operational terms, data orchestration collects isolated data from different locations within a company, processes it into unified and usable intelligence, and activates it wherever it’s needed to:
Support Any Business Team with Any Data
Data orchestration eliminates the need for manual data preparation, allowing any business team to extract and put data into action in real-time. Business teams can solve their own problems with data and automation.
Enhance Strategic Alignment + Collaboration
Without data orchestration, isolated projects can be misaligned with business goals and each other. With data orchestration, every solution is an expression of the same data, facilitating strategic alignments across business goals and projects.
Reduce Costs + Friction
Ad-hoc data engineering projects are hard to scope, expensive to iterate, and often produce brittle results. Data orchestration makes it possible to make incremental investments and improvements with quick feedback loops at a lower cost.
The Billion Dollar Intelligence Layer
The operational benefits above, while compelling, don’t sound very different from the ye olde streamlining of the mid aughts. We’ve been talking about data for a long time, and it always promises to do the same thing— provide an empirical foundation for work.
So why is now different? The billion dollar idea here isn’t exclusively operational. What AI makes possible now is rich, rapid intelligence at scale. Gone are the days of the “data-driven.” Here is the era of the “intelligence-activated.”
AI can unlock a tremendous value in stagnant data. General purpose language models can be adapted very, very quickly to extract specific data signals from text, and even from structured data like customer profiles and event streams. This level of processing has never been possible before. The depth of insight now available at scale is genuinely revolutionary for work.
Until now, most data reflected behavior out of context. You could know who clicked what and take your best guess as to why. You could know why someone submitted a support ticket but not how their experience with an agent impacted their attitude about your company. AI takes us from having some retroactive visibility into consumer behavior to using real-time models of consumer intelligence to drive core business processes.
In the next few years, the world will collectively experience the implications. Product and service development will be different. Workflows and organizational structures will change. The internet rapidly democratized access to information; now AI is rapidly democratizing access to intelligence. Organizations investing in data orchestration now will be the first to harness the network effects of this democratization.