Agents will do more of our analysis. Some will answer questions we ask; others will ask and answer their own questions while pursuing a goal. Tristan Handy calls the second kind agent-initiated analytics, and suggests those queries could outnumber human-initiated ones by a hundred to one within three years.

Agent-initiated analytics is exciting because the agent can act on what it finds. It can power a self-learning loop: investigate, act, measure and use what it learns to decide what to do next. Companies able to run these loops across their operations could move much faster, learn more systematically and adapt more effectively as customers and markets change.

Autonomy is what makes this so consequential. Within agreed objectives and constraints, the agent can take the next step without waiting for a person to interpret each result and authorize each action. People still decide where that autonomy is appropriate. To earn it, the agent must use the right data and definitions, distinguish an effect from noise, and recognize when the evidence is too weak to act.

That is why I think we need to build agents that author context as well as consume it. As an agent works, it should record what it tried and why, and connect that account to what actually ran and what happened next. That gives the same agent, or a different one, a history it can learn from. The challenge is carrying the meaning of that context from creation to use.

What agents that pursue goals unlock

Autonomous agents can run these loops in parallel across campaigns, features and customer experiences, with each decision building on previous attempts. By learning, I mean retaining what happened and using it to improve subsequent decisions. That could change the agent’s operating policy, the evidence it consults or, in some systems, the model’s weights. An enterprise could build and run thousands of these loops, making systematic learning part of how it operates.

Advertising gives us a precedent for handing the optimization of a loop to a computer. Google made Performance Max generally available in November 2021; Meta introduced Advantage+ Shopping in 2022. The advertiser sets a budget and objective; the system handles ongoing optimization. Google describes budgets and bids being optimized across channels in real time. Machines can make these repetitive decisions far faster than people, applying a policy consistently at a scale no team could manage by hand. Meta told investors in July that its Advantage+ end-to-end solutions had passed a $75 billion annual revenue run-rate, an indication of how widely advertisers have adopted this way of working.

The opportunity I see with agents is to extend that autonomy to much more of the work. An agent can formulate a new question, combine tools and change the intervention itself: create a different offer, build a landing page or investigate why a segment responds differently. More of the work between setting a goal and pursuing it can happen without a human specifying steps in advance.

The strongest candidates for autonomous agents have measurable goals, controllable actions and feedback that can distinguish improvement from noise, arriving soon enough to inform the next decision. Andrej Karpathy’s rule, “Software 2.0 easily automates what you can verify,” is critical here. For a business loop, the challenge is verifying progress toward the actual goal, and then diagnosing why that happened. That is a data and analytics challenge.

What this looks like in marketing and product development

Consider a mid-sized e-commerce company. Channel managers optimize within their channels; the CMO periodically reallocates budget between them. Analysis informs those decisions, but moving from a finding to a coordinated change takes work.

Now imagine an agent tasked with growing incremental revenue at a target incremental cost of acquisition. It has constraints: brand requirements, a minimum budget per channel and a maximum daily change. It monitors performance hourly and reallocates when the evidence justifies it. It commissions incrementality tests when attribution alone cannot answer the question. Within those constraints, the agent can act on its analysis without waiting for someone to review it.

The components already exist, with different levels of autonomy. Haus describes execution after approval; Recast exposes planning and optimization tools through APIs; Sellforte offers autonomous and assisted media buying. The harder question is when the evidence is reliable enough to let the system move money autonomously.

Product development offers a similar possibility. A team ships a feature to move a metric, runs an experiment and decides whether to roll it out. In September, AWS and LaunchDarkly published a reference implementation connecting code changes, experiments and guarded rollout, including a scheduled agent that can advance a winning treatment autonomously. The demonstration uses synthetic traffic: it shows that the steps can be connected, but does not establish how quickly a real business can learn.

Automatic safeguards are also available. Datadog’s guardrail metrics can trigger rollback when a release breaches a threshold. But detecting harm and establishing a commercial benefit are different jobs. The Booking.com conference recap draws the lesson from Marcel Toben’s talk that cheaper software changes make rigorous evaluation more important. Knowing whether a feature worked still takes a correctly designed experiment, enough traffic, appropriate guardrails and a valid stopping rule.

What it costs when the analysis is wrong

Giving an agent the levers makes an analytical mistake expensive. The same mistaken interpretation can drive repeated changes across campaigns, features or customers before someone notices.

Zillow’s experience shows how costly it can be to discover that a forecast is unreliable after capital has already been committed. In Q3 2021 it recorded a $304 million inventory write-down. Forecasting uncertainty and renovation and resale capacity constraints contributed to its difficulties. The company announced in November that it would wind down Zillow Offers. An observable outcome does not guarantee timely feedback.

Feedback can also reward the wrong behavior. A marketing agent optimizing a biased attribution metric might report success while generating little incremental business. Measured’s Trevor Testwuide puts it sharply: “Automation does not remove attribution bias. It industrializes it.” The agent can improve its assigned measure and still miss the business goal.

These problems require different remedies. Better records alone cannot guarantee timely feedback or turn a biased metric into a valid measure of success. But without reliable records of actions and outcomes, the agent also lacks the evidence to discover what went wrong.

What an agent needs in order to learn

For agents to learn reliably from their actions, the system needs to preserve what they tried, what actually ran and what happened next, with enough context to connect and interpret those records.

A marketing agent’s work leaves three kinds of data behind:

1. The decision and its context: which budget moved, when, and which reading of which numbers justified it.

2. The campaign that actually ran: its creative, landing page, offer and version.

3. The customer behavior that followed: who saw it, clicked and converted.

A product agent needs the equivalent: the hypothesis and chosen cohort, the feature actually delivered, and how users engaged with it. The agent can supply its intention and rationale. Instrumentation should independently record execution and outcomes. A convincing account of what the agent meant to do is not evidence that it happened.

Suppose the marketing agent changes a campaign’s offer as well as its budget. Conversions rise. If the record contains only channel spend and total conversions, the next analysis may credit the budget move for a change caused by the offer. Recording both changes against the campaign version makes that competing explanation visible. An experiment or credible causal model is still needed to separate their effects.

The pieces often exist in different systems: an agent trace, a campaign or feature record, and customer events. Learning needs reliable connections between them, with shared identifiers, versions and definitions. Outcomes arrive later, and one intervention can affect many customers.

Observability tools such as LangSmith and Langfuse support tracing, evaluations, feedback and additional context. They provide useful foundations, but the business connection between a budget decision, a campaign version and customer behavior still has to be designed and maintained across systems.

The meaning must survive too. Anthropic reports that its analytics agent achieved no more than 21% accuracy without skills and over 95% with them on its internal evaluations. The skills supplied procedural guidance, retrieval, governed data and validation. Pointing the agent at existing SQL files alone improved accuracy by less than a percentage point. The account also describes offline accuracy falling from 95% toward 65% within a month as context became stale. Maintained context mattered as much as access to data.

Spotify’s data assistant team and DoorDash’s Vera team describe the importance of domain expertise, context and evaluations too. Their work supports a practical conclusion: reliable analysis requires more than giving a model a database connection. It does not require a domain expert to sit beside the agent for every query.

For new data we control, we can reduce the reconstruction work by capturing meaning when the data is created. Ananth Packkildurai describes early binding as capturing semantic intent at production through data contracts. He also stresses that this alone is insufficient: context must survive downstream transformations, and sources outside our control may need later contextualization. For autonomous agents, my proposal is to make authoring enterprise context part of the work itself.

Return to the marketing agent. When it proposes a budget move, it can record the channels, amounts, campaign version, evidence and hypothesis. The execution system can record whether the move actually happened. Customer events can carry identifiers that connect them to the campaign and offer customers saw, using agreed definitions of a click or conversion. A week later, an agent investigating the result can recover that history instead of guessing what changed.

As agents take responsibility for more work, they can author more of the context that makes it understandable. If they do this reliably, each campaign, feature and experiment leaves a usable history for other agents, including those introduced later. The more work agents do this way, the easier it becomes for their successors to learn from what has already been tried.

They need not author every record themselves: historical data and independently measured transactions remain useful evidence. But the campaign version, metric definitions, evidence and assumptions must remain connected and interpretable as records move between systems and agents. Definitions still need owners, validation and maintenance; an assumption must remain distinguishable from an observed outcome.

People therefore remain responsible for the standards of evidence and for evaluating the system. They can inspect exceptions without interpreting every routine result. When the evidence is inconclusive, the correct decision may be to keep the current policy, gather more data or escalate. An agent that knows when to wait is more useful than one that always finds a reason to act.

I must note that I am not a neutral observer: this is the problem we are working on at Snowplow. Our focus is agents that build applications, deliver marketing and shape user experiences in real time. We want the context they author to remain connected to what actually ran and how customers behaved, ready for the same agent reviewing yesterday’s campaign or a different agent designing next month’s experience.

Context must retain its meaning from the moment an agent authors it to the moment that agent, or another agent, consumes it to learn. Carrying context from creation to use lets future decisions build on what earlier agents tried and what actually happened.

With that foundation, autonomous agents can act quickly, learn systematically and pass those lessons on, helping the business become more agile and effective with each cycle.

People will still read analysis: to learn what agents found and to check how they reached their conclusions. What that does to the people whose profession is analysis is the next post.

I ended my last post with the wrong question.

In that post, I argued that traditional multi-tenanted SaaS applications could not rearchitect on cloud data platforms because they could not work on a data layer they did not control: the integration costs were simply too high. The next generation of operational systems will be agentic, and a set of agents working in one tenant can learn to use the data in the customer’s own cloud data platform without worrying that the data in the next tenant looks completely different. That is what has created the opportunity for the cloud data platforms to become the foundation the next generation of operational systems is built on.

The question I asked was “does decision gravity follow data gravity?” Will the next generation of agentic applications run natively on the cloud data platforms? I framed it as a question about compute: if I build an agent that uses my data in Snowflake, do I build it with CoCo (formerly Cortex Code), or do I just as easily reach for Claude Code, OpenAI Codex or some other agent-and-harness combination, given that any of them can help themselves to that enterprise context via MCP?

It was reading Brett Queener’s excellent The Harness, the Horse or the Hay that made me realize it was the wrong question. Queener imposes a devastating simplicity on the agentic application stack. He boils it down to three types of solution:

• The horse. This is the LLM.

• The harness. This is the agentic application. A harness is built for a specific persona or business function, and it needs to do everything that persona needs. Harvey for lawyers and Cursor for software developers are both harnesses (they are two of Queener’s own examples). Harnesses that are not built for a specific specialism end up being built and owned by the horse: that is what we are seeing with the spread of Claude Cowork and ChatGPT Work across all generic knowledge work. It matters here because it leaves the specialist harnesses, and whatever they are built on, as the ground the platforms could still take.

• The hay. This is everything else: ingestion, serving, memory, evals, observability, and the data platforms themselves. Queener puts data platforms like Databricks and Snowflake here.

So the question is: who will own the harnesses the next generation of enterprise business applications are built on? Do the cloud data platforms have the opportunity to become harnesses for one or more business functions as they work their way up into the application layer? Or could they become something more useful, a meta-harness: the thing other companies, including their own customers, build their business-focused harnesses on top of?

If they can, the compute and the billing that goes with it will follow. I believe that they have the opportunity, but that today they are missing at least one primitive they need to seize it, and probably more: the ability for agents to create enterprise context.

The naive argument for decision gravity following data gravity

Agentic applications are loops. The agent has a goal. To pursue it, it observes, orients, decides and acts, and then does it again with what it has learned. A marketing agent that shipped an offer to 40,000 lapsed subscribers and saw 2% of them come back should behave differently tomorrow, and a support agent allowed to refund anything under $50 without escalation should learn which of those refunds come back as chargebacks.

Enterprise context is the pre-requisite for these loops. It powers the observe and orient phases, because it gives agents broad and deep visibility into what has changed across the enterprise. And it should enable the learning, because it ought to be possible to read from the data every decision made in the past along with what actually happened.

The speed with which these applications learn will be critical. Learning compounds, and as a buyer you would be mad to buy a solution that learns more slowly than the alternative. Putting the decision next to the data is one way to make the loop run faster. So, the argument goes, rather than bring your enterprise context to your agent, bring your agent to your enterprise context and build it natively in Snowflake, Databricks, BigQuery, Fabric or ClickHouse. That is the platforms’ long-running pitch, “bring AI to your data”, and it makes a lot of sense.

To learn effectively, agents need to be able to create enterprise context

There are two overlapping problems for the naive argument.

The first is that important context about the past is not in the enterprise data platform. The best example is the decision trace: the record of who decided what, on what grounds, with which exceptions. Jaya Gupta and Ashu Garg at Foundation Capital set out both the gap and the value of this data in their excellent Context graphs: AI’s trillion-dollar opportunity. Those traces, in their words, live today “in Slack threads, deal desk conversations, escalation calls, and people’s heads”, and “by the time data lands in Snowflake, the decision context is gone.”

The second is that effective agents are going to change the future of the organizations they run in, quite literally transforming the context in which they operate. That new context needs to be captured and used to power learning. Agents need to be able to create enterprise context. The ELT pipelines, dbt jobs and human-curated semantic layers that today’s data stacks are built on are not fit for this, for two reasons. They record the state of source systems, not decisions or the grounds for them: a Salesforce sync tells you the opportunity closed, not why the discount was approved. And they change on a human release cycle, one pull request at a time, while agents will be deciding and learning continuously. Nor will throwing hundreds of agents at the data exhaust and hoping they assemble it work.

What does “create enterprise context” mean in practice? Writing data is not the hard part: an INSERT over MCP into Snowflake is a solved problem. Take the agent that shipped the offer to 40,000 lapsed subscribers. For the next decision in the business to be better for it, human or agent, in marketing or outside it, the record has to say who the 40,000 were and by what definition; what the offer was and why it was chosen over the alternatives; what happened, against which baseline; and what the agent concluded. The audience definition has to reconcile with the one the finance team uses for the same customers. The outcome has to be attributed in a way the sales agent will not contradict next quarter. That is an editorial act on the enterprise’s shared description of itself, and there is no governed path for it today, in the platforms or in the applications.

That record, not the underlying data, is the asset with gravity: the governed, continuously written account of what the business knew, decided and learned, in terms the whole business shares. I am going to call the pull it exerts context gravity, to distinguish it from data gravity, which MCP has made weak, and from decision gravity, which is the prize rather than the contest. Agents will end up where their context can be written, not merely where the data can be read.

What the Foundation Capital argument gets right, but why I think it is wrong

For Gupta and Garg, the fact that the cloud data platforms are missing some of the most valuable context agents need, and have no clear path to acquiring it, means that decision gravity will not follow data gravity. Instead, a new breed of agentic startups will be where the decisions take place, capturing the traces “at decision time—not after the fact via ETL, but in the moment, as a first-class record”. Their examples are Regie in sales engagement, Maximor in the finance close and PlayerZero in L2/L3 support. Sierra is the clearest case in the wild: last December it announced an Agent Data Platform with SiriusXM to give its customer-service agent “the context to know not just what a listener is asking for today, but what will make them smile tomorrow”. No warehouse is named. Sierra is building its own data layer, and what its agent learns about SiriusXM’s listeners stays inside it.

I don’t buy the argument. Agents, and therefore their harnesses, need access to all the data required to do their jobs. If that data were siloed in the domain the agent serves, the private-context model would be workable. But enterprises are interconnected systems: things that happen in one domain have consequences in others. Foundation Capital’s own illustrative workflow concedes this, with an agent checking ARR in Salesforce, escalations in Zendesk and churn signals in Slack: the context that matters is cross-system by their own account, and a graph inside one vendor’s execution path sees one role’s slice of it. One of the strengths of the cloud data platforms is that, set up right, every domain contributes its own data and has visibility of everyone else’s.

Suppose the private-context model wins. Sierra builds a context store for the support agent; the marketing harness builds one for the marketing agent; the sales and finance harnesses each build their own. Five agents, five private world-models of the same customers, each getting better in isolation, none of them correcting the others. That is the SaaS data sprawl of my last post, recreated in the new agentic paradigm. Twenty years ago every department bought the tool with the best UX for its workflow and the business’s data fragmented across a hundred vendors; the warehouse was supposed to end it. The agentic version is worse, because what fragments this time is not the data but the learned judgment.

So at minimum the harness will need read access to the data platform. But what sort of enterprise is going to give a vendor’s agent access to its data and then not own the resulting enterprise context, all those decisions and all that learning, because it lives in the vendor’s harness? I think they will insist on it being written back. If it isn’t, they don’t own their own intelligence, or their own data.

That is a prediction rather than a finding. In the deals I see, buyers have started refusing software that will not expose its data and functions over MCP; in about a year it has become a procurement line item, and it applies to the agentic vendors as much as to the incumbents. That is a demand to read data and call functions. The demand for write-back, for the vendor’s agent to record what it learned in the customer’s shared model rather than in the vendor’s private one, is complementary and critical: a business cannot run on N agents each learning privately from N divergent pictures of the same customers. Foundation Capital have identified the asset correctly. But they are wrong about who gets to keep it.

What we can learn from aircraft engineering

There is a much older industry that I think is instructive. An airliner is on a stand three thousand miles from its airline’s own hangar. A contractor’s engineer, employed by a maintenance organization the airline has never dealt with, replaces a hydraulic pump. Before the aircraft flies, the engineer records what was found, what was done and why in the aircraft’s technical log, and signs it. The log stays with the aircraft, not the contractor. Under EASA’s technical-log rules (Part-M M.A.306) and FAA Part 121 (§121.709), the aircraft may not be operated after maintenance without a signed airworthiness release or log entry. And the next engineer to open a panel, at another airport, employed by another company, reads that entry before deciding what to do.

The record belongs to the asset’s owner, not to whoever did the work. Writing to it is governed: there is a format, an authorization and a signature. And the record is the mechanism by which the next decision is better than it would otherwise have been. Contractors are interchangeable precisely because the log is shared. An airline that let each contractor keep a private log of what it had done to the fleet would not know the state of its own aircraft. Nobody proposes the private-log model for aircraft. It is exactly what the harness vendors are proposing for the enterprise.

So what are the missing primitives the cloud data platforms need to provide?

Two, one for each half of the loop.

The ability for agents to create enterprise context. A governed write path by which an agent, including one the platform did not build and does not bill for, records what it knew, decided and learned, in the customer’s shared model of the business rather than in a private store. Governed the way the technical log is governed: a schema for the record, versioning so that the definition of a customer or an outcome can evolve without breaking the agents that depend on it, and an authorization so that a marketing agent cannot silently redefine what the finance team means by revenue. This is a primitive the data platforms are missing, and it is the one that decides whether they are the meta-harness or the hay.

The ability to use that context to power systematic learning. Decisions joined to outcomes, searchable as precedent, and a grading and feedback substrate that a harness builder can use rather than build privately: the machinery by which a graded outcome changes the agent’s future behavior without an engineer deciding what to change. Foundation Capital are right that this is where the value compounds; the question is whether it compounds in one vendor’s silo or in the enterprise’s shared context. How agents actually learn, and what the platforms owe the people building harnesses on them, is an area I plan to explore in future posts.

The platforms are moving in this direction, and it is worth being exact about how far. Snowflake now describes its ambition as being “the control plane for the agentic enterprise” and says Snowflake Intelligence “adapts over time by learning individual preferences and workflows”. Databricks announced in June a managed memory service in which agents “manage their own context, session history, and persist them across sessions”, says of agents’ reasoning traces and memory writes that “that data should be governed in the Lakehouse alongside the rest of your data”, and has added to Unity Catalog “all agents, models, MCPs, Skills, and external agents”. Memory, write-back and a registry of third-party agents are the parts. The question is whom they are for. A memory service for the platform’s own agents is a feature. A write path that a rival’s harness is invited to use, in the customer’s shared model, under the customer’s governance, is a meta-harness. The test of intent is whether it works for an agent the platform does not bill for, because that is the case that does not obviously increase consumption, and it is the case the customer needs.

I should say what my own interest is. At Snowplow we have spent more than a decade learning how hard it is to create behavioral context well: to write a description of what a customer did that is accurate, versioned, governed and still means the same thing to the marketing team, the product team and the model that consumes it. We are now working on what it takes to write that context back for agents, and the write-back problem for decisions has the same shape as the creation problem we started with. The hard part is agreeing what the row means: which customers count as lapsed, which outcome counts as a return, and whether finance uses the same definitions.

Does decision gravity follow data gravity? Maybe, but not necessarily

Will the data platforms get to own the meta-harness on which the next generation of agentic, vertical-focused apps are built? Maybe, but only if they build all the required capabilities. Any agent can read the data over MCP, and the learning loop is pulling context out of the data platform toward the harness, which may or may not be running natively in the data platform. Decision gravity follows context gravity, and context gravity accrues to whoever provides the governed, shared path for writing what the agent learns on behalf of the business. The cloud data platforms are the natural home for that path, because they already hold the data the context describes and because the customer will not tolerate a private log per vendor. But they have to build it, and Sierra is not waiting; nor are Foundation Capital’s companies. If the platforms build a write path only for their own agents, the harnesses will keep their own logs, the enterprise will fragment again, and the platforms will have won the data and lost the decisions. There is a lot to play for.

In 2022, Matt Bornstein, Jennifer Li and Martin Casado of a16z updated their much-cited map of modern data infrastructure and made a bold claim they called the data platform hypothesis. The claim: the cloud data platforms (Snowflake, Databricks, BigQuery) were becoming platforms in the proper sense of the word, things that other developers build on. “A single customer may buy and build many applications on top of one core data system,” they wrote. And then the line that matters: “We’re even starting to see traditional enterprise systems, like financial or product analytics, being rebuilt with a ‘warehouse-native’ architecture.” The point was driven home with the following diagram:

Source: a16z, Emerging Architectures for Modern Data Infrastructure (2022 update)

That was the maximal version of the modern data stack dream. Not just that analytics would consolidate onto the warehouse, but that business applications i.e. operational systems, the software companies actually run on, would rearchitect around it. The platform vendors said the quiet part out loud. “We want to make Snowflake the iPhone of data applications,” Benoît Dageville told ITPro in June 2023, the same month Databricks launched Lakehouse Apps.

It did not happen. What we saw was SaaS vendors build improved integrations with the cloud vendors - with clearly documented approaches for users to export data from the application into the data platform, and ingest information from the data platform into the application. But bar a handful of notable exceptions (Snowplow being one of them), the vast majority of business applications preserved their own backend systems and with it their own version of the data.

I think the claim is finally going to come true and the reason is AI. In this piece I’m going to explain why. However, I don’t think that necessarily means that the agents and agentic systems that do that operational work are necessarily going to run natively in the data platforms. Whether that “decision gravity” (to use Alex Dean’s phrase) follows data gravity and accrues to the data platforms is something I’ll explore further in the next post.

A (thankfully brief) history of SaaS

If we want to understand what might upend the current status quo we need to start by understanding the status quo that AI is steadily disrupting.

SaaS as we know it was invented by two companies: NetLedger (which became NetSuite), which launched a general ledger on the web in 1999, followed a year later by Salesforce, which launched its SaaS CRM in February 2000.

The first two SaaS solutions offered broad propositions: general ledger and CRM, competing in two large traditional software markets. They were followed by an explosion of new SaaS vendors, VC-backed, offering increasingly narrower propositions, focused on helping specific business buyers solve specific (often one single) problems or workflows. We need to understand why - because as I will argue, this narrowing of SaaS propositions is what created the data fragmentation problem that the data platforms have been so effective in solving, and has left many SaaS solutions vulnerable to disruption by AI today.

Why did SaaS solutions become so narrow and specialized?

SaaS drove a number of significant changes in the way software was built and sold:

• SaaS solutions were bought by the lines of business, rather than IT. Because the lines of business did not need IT to deliver the solution, they were free to buy what they wanted, with their own budget. This was a significant shift from traditional software, where if the procurement was not led by IT they were at least a significant secondary stakeholder

• Ease of use, time to value, user experience and ease of buying became the moats

SaaS was dominated by VC backed startups, that ruthlessly focused on winning their categories. An effective strategy to winning a category was to define a very narrow category, ideally one that no one has focused relentlessly on, and then build the most compelling (fastest time to value, delightful UX) solution for it. Being a compelling CRM requires an enormous surface of functionality. Being a great email marketing tool does not. A narrower category means a faster build, a clearer pitch to a more specific buyer, and a much better chance of being the winner that takes most of it. Tomasz Tunguz described the result in 2015: teams now “purchase best-of-breed point solutions, try them, and quickly cycle through all the different offerings”, selecting “precisely the products that offer the features that meet their specific needs.”

The slices got thinner, decade by decade. Mailchimp (2001) carved email off the marketing suite; Klaviyo (2012) re-carved it for online stores specifically. Zendesk (2007) was just tickets, Expensify (2008) just expense reports, Calendly (2013) just the scheduling link, Gong (2015) just recording and mining sales calls. Each is a real business, some of them very large, each built around one tidy workflow for one person.

The clearest record we have of where this ends is Scott Brinker’s marketing technology landscape: about 150 vendors in 2011, 15,505 by 2026, a hundredfold increase in one function’s tooling in fifteen years.

Products on Scott Brinker’s annual martech landscape, 2011-2026. Source: chiefmartec.com / martechmap.com (Scott Brinker with Frans Riemersma)

The trouble with SaaS was it drove widespread data fragmentation

The price of a thousand tools, each optimised around helping a different team member do their job, is that the data ends up fragmented across all of them. The conversations live in Gong, the forecast in Clari, the engagement history in Klaviyo, the tickets in Zendesk. SaaS optimized UX at the expense of data.

Marketing, the function that bought most enthusiastically, felt it first and worst, and an entire category was invented to patch the damage: the customer data platform, whose founding purpose was to reassemble the customer that a hundred marketing tools had shredded.

The warehouse to the rescue

This is the problem the cloud data warehouses walked into, with an offer that was very hard to argue with: bring it all here. Unify it. Build the single customer view, the single source of truth, and use it for analysis first and operations next. It worked; that consolidation is what we spent a decade calling data gravity, and it is why the modern data stack existed at all.

The data platform hypothesis was the logical endpoint: if all the data is here, why shouldn’t the applications be? But it was not uncontested. Casado himself, on the Great Data Debate podcast in 2020, warned that “we look at an architecture, and we’re like, oh, that can do all of these things, therefore it will be pushed into service to do all of these things. And that’s just not how technology evolves.” And for five years it seemed that he was right.

The trouble with rearchitecting SaaS applications on the data platform was that building an application on a data foundation that you do not control is incredibly difficult. Every customer’s warehouse is different: different data coverage, different schemas, different semantics, different quality. The attraction of building an application on the data warehouse is that you can leverage all the data, but the cost of using all that data is prohibitively high: it takes real integration work to map that sprawl into a standard schema that an application can work with, and graceful fallback where parts of that data are necessarily missing. The cost scales with the value, so there is never a point of pay-off.

Agents fundamentally change the economics of building applications on the modern data stack

When software changes from a UX to make it easier for a human to perform a workflow, to an agent that will do whatever it takes to drive a business goal, that cost benefit analysis inverts.

Previously, SaaS systems optimized UX at the expense of data fragmentation. Agents care little for UX, and a lot about good data. It is a struggle to build traditional multi-tenanted SaaS applications on top of data layers where the schema, semantics, quality and ownership vary from tenant to tenant. But an agent working in one tenant that has access to accurate metadata (context) can navigate that data effectively, and does not care one bit if another agent working in another tenant is working with a data set that looks completely different. Agents do not make the integration cost zero, but they dramatically reduce it to the point it is no longer a barrier.

And if the value of a SaaS application is in providing a delightful UX for a human completing a narrow task, agents drain that moat pretty quickly. Enterprise companies and marketing teams in particular might be very happy to buy 100 different point solutions to optimize 100s of different workflows for lots of different people - but agents would much rather have access to a finite set of broad tools that enable them to take all the actions that they need.

So the conditions are ripe for a new breed of agentic application that runs directly on the cloud data platforms.

And we are now seeing a new generation of agentic application emerge on the cloud data platforms

In June 2026 Databricks launched CustomerLake, a customer data platform rebuilt natively on the lakehouse, with campaign agents that build audiences and run campaigns directly on the customer’s own data foundation. Hightouch’s AI Decisioning lets a marketer “assign a goal like ‘drive customers who buy in-store to download our app and order ahead’” and hands execution to agents that run on the customer’s warehouse; PetSmart is using it across a loyalty programme of more than 70 million members.

Nor is the pattern confined to marketing. Wolfspeed, the semiconductor manufacturer, has deployed dozens of AI agents across manufacturing, quality, supply chain and finance on Snowflake, and its CIO describes exactly the arc this piece has been tracing: first consolidating “200+ SQL reporting silos into a unified, governed data ecosystem”, then putting agents on top, “troubleshooting equipment issues in two minutes instead of two hours”. United Rentals is building agents on Snowflake “to accelerate sales growth and improve fleet availability” across more than 1,600 locations. 7-Eleven built its maintenance technician assistant on Databricks Agent Bricks, over its own documents in Unity Catalog. And enterprises are building fully bespoke agents too: AIG has built underwriting agents with Anthropic over its own data estate, compressing underwriting review timelines by more than five times.

A CDP on the lakehouse, goal-driven marketing agents on the warehouse, factory and sales agents on Snowflake, underwriting agents over the enterprise data estate. That is the “warehouse-native” rebuild Bornstein, Li and Casado described in 2022, arriving four years late and wearing an agent’s badge.

But it is not clear whose agents these will be, or where they will run

The argument that operational systems would rearchitect around the cloud data platforms was hugely exciting for the cloud data platforms and their investors, because the assumption was that meant all the compute for those applications would run on those data platforms.

It is not obvious that that is the case though. If I want to build an agentic application on my data in Snowflake, I could leverage Cortex. But I could also leverage Claude, OpenAI, AWS or many other providers, and enable that agent access to the data via MCP. So it is not a given that “Decision Gravity” will follow “Data Gravity”. That is something I will explore in a lot more detail in my next post.

In my last piece I argued that for twenty years we in data were wrong. We believed that culture and data literacy were what held back data democratization. Then AI arrived as a new interface onto the same data, democratization happened anyway, and it happened just as fast in the organizations that had never run a literacy program. It was never culture. It was that the tooling was not usable by anyone who was not a data person.

So the gap I have spent my career worrying about, between the data people and everyone else, has stopped being a gap in access. Anybody can now ask a question of the data and get an answer back in seconds. What has not changed is that using data well is difficult, and the people doing it now were never taught the ways it goes wrong. They are exposed to a whole category of mistakes that neither they nor the people they present to can easily spot, because a wrong answer arrives looking exactly like a right one.

There is still a job here for culture, and I want to be precise about what it is, because it is much smaller and more specific than the thing we spent two decades demanding.

Mostly, though, this piece is about what to do about it. The mistake we made with data was to decide that the problem lay with the people using it, and that the job was therefore to fix them rather than to empower them with better technology. We are at risk of making that exact mistake again with AI. By the end of this you will have a single question you can ask of your own AI plans this quarter that tells you whether you are making it.

The gap is now between answers you can defend and answers you cannot

The gap I described in the last piece is between two groups of people: those with real analytics experience and expertise, and those without it. For twenty years it decided whether you could get an answer out of the data at all. With the expertise, or the attention of someone who had it, you got answers. Without it, you got a dashboard and a training course.

Everyone can get an answer now. What twenty years of experience buys you today is not the answer, it is the ability to tell whether the answer is right. And on the slide, in the meeting, an answer produced by someone with that experience is indistinguishable from one produced by someone without it.

What happened when I asked about our sales calls

A few months ago I asked Claude a question about our Gong data. I wanted to know how often a particular selling strategy showed up in the calls on our open opportunities. Worth saying what that requires, because it is not universal yet: a warehouse with the Gong data in it, and Claude given access to query that warehouse directly. If your business data is not somewhere an agent can reach it, none of what follows has happened to you yet.

Claude came back quickly and confidently with a percentage. Fifty-seven percent! Two significant figures.

I pushed on it and the number fell apart. It had looked at a handful of opportunities and reported the proportion as though it had checked them all. The precision was spurious, and the spurious precision is what made it convincing: nobody reports 57% unless they have counted something.

What I ended up asking for was a spreadsheet: one row per opportunity, the categorization for each, and a link to the section of the Gong call that justified it. Then I could see it really had been through every account, and that the arithmetic was arithmetic rather than an impression.

It is not that I am unusually rigorous. I knew where the number was going: into a meeting with colleagues who would ask me how I got it. Twenty years in data has left me with the reflex of being ready to defend any number I put in front of anyone.

Most people asking these questions have not spent their careers in data and have never needed that reflex. The number goes into a slide, the slide goes into a meeting, and the room takes it.

It is worth being clear about which part of that went wrong. My question was fine: how often does this selling strategy appear in the calls on our open opportunities. What broke was the answer, and I caught it because a suspiciously round-sounding 57% is the specific thing twenty years teaches you to poke at. There is a second failure that is harder to catch, where the question itself is wrong and the answer to it is entirely correct. Nothing about that one looks wrong, because nothing about it is wrong except the question.

“Claude says”

There is a phrase I now hear most weeks, and I would guess you do too. “Claude says.” Sometimes it is Claude, sometimes another name, but the sentence is always built the same way: a number arrives in a meeting or a Slack thread attributed to an AI, and that is the end of the discussion. Nobody asks which opportunities were included, over what period, or how the behavior was identified. Six months ago somebody would have asked at least one of those.

The mistakes underneath are not new ones. The wrong dataset. A sample reported as though it were the population, which is exactly what happened to me. A correlation narrated as a cause. An answer to a question slightly different from the one asked. What is new is that they now arrive dozens of times a day, within seconds, and pass nobody whose job it was to catch them.

Writing the query was only the visible half of what an analyst did. The other half was saying “that is a vanity metric”, or “you have the causation backwards”, or “we cannot answer that with the data we have.” No BI tool ever did that half. A person did it, and in most of these workflows there is no longer a person in the path.

The really valuable thing a good analyst does is ask a good question. The technical work of answering it is the easier half, and AI has shown that it can do that half well, even if, as with my 57%, it sometimes does it wrong. I have yet to see AI do a good job of asking the right question.

So there are two problems here, not one, and they want different answers.

The first is that AI lets people produce analysis that is wrong in ways they cannot see. That is the 57%: a good question, a bad answer, and nobody in the room equipped to notice.

The second is that AI lets people ask the wrong question and get a flawless answer to it. Nothing about that output looks wrong, because nothing about it is wrong except the question.

I think I know what to do about the first. I am much less sure about the second, and I will say so plainly when I get there.

Problem one: teaching people to be more skeptical will not fix it

The obvious response is the one I reached for first, and the one I keep hearing: teach people to be skeptical. Get them to ask the AI to show its working. Challenge it step by step. Treat AI literacy the way we treated data literacy.

I no longer think that works as the primary answer, and the reason is uncomfortable.

Ethan Mollick has described the problem more sharply than anyone: “AI is often most useful where we’re already expert enough to spot its mistakes, yet least helpful in the deep work that made us experts in the first place.” On why we do not catch the errors: “Hallucinations are therefore very hard to spot, and research suggests that people don’t even try, ‘falling asleep at the wheel’ and not paying attention.”

There is evidence for the mechanism. A 2025 study from Microsoft Research and Carnegie Mellon surveyed 319 knowledge workers about 936 real uses of generative AI and found that “higher confidence in GenAI is associated with less critical thinking, while higher self-confidence is associated with more critical thinking.”

And Stancil closes the last escape route, in the best thing I have read on this: “There’s no way to know if a chart ‘works’ unless you understand how it was built; the validity of the external product depends on the validity of its internals.” Software can be tested by running it. Analysis cannot. “All you can do is either read through the code, line by tedious line, or recreate the whole thing yourself. And if you have to do that, what exactly are we automating here?”

So “ask it to show its working and challenge each step” asks someone who is not an analyst to evaluate work they could not have produced, using judgement they were never given, on output engineered to look careful. It puts the entire burden back on the person, which is precisely what we did for twenty years.

That is the same prescription we wrote for twenty years, with “AI literacy” in the place where “data literacy” used to be.

Problem one, solved: put the burden in the system rather than on the person

The alternative is to stop asking the person to carry what the system can carry.

Anthropic’s data team published an account of their own internal analytics in June, and it is the most useful thing available because it comes with numbers: “Without skills, Claude’s ability to answer analytics questions accurately didn’t exceed 21% on our evals. Adding skills gets these numbers consistently above 95% in aggregate.”

Accuracy went from 21% to 95%. Every point of that came from work done to the system: definitions written down, context structured, rules encoded. Nobody ran a training course for the people asking the questions.

They also published two failed experiments that are more instructive than the headline. First, they gave the agent grep access to thousands of files of existing dashboard, transformation and analyst SQL, and verified it was reading them: “Accuracy moved by less than a point in either direction. The information was there, the agent saw it, and it still didn’t use it. That single experiment told us our bottleneck wasn’t access to prior work, it was structure.” Second, they tried having an LLM write the metric definitions from raw tables and query logs: “It produced plausible-looking definitions that encoded the very ambiguities we were trying to eliminate.”

You cannot retrieve your way out of this, and you cannot ask the model to do it for you. Somebody has to write down what the business means. Their name for the thing nobody does is business context, “the layer most teams skip, and the one we underrated the longest. An agent that doesn’t understand your business will answer what the user asked, but not what they meant.” Nor is this one lab’s idiosyncrasy: OpenAI’s data platform team, which takes the architecturally opposite approach to the same problem, opens its own account with a section headed “Context is everything.”

Two things follow, and both belong in the system rather than in a training deck.

The agent needs an explicit model of how the business works. Almost every question a stakeholder asks is a “why” question, and answering one requires knowing what drives what. Abhi Sivasailam and Trevor Fox make the point in their work on metric trees, which predates agents entirely: data models give you “the semantics for discourse and analysis” but “fall short in providing the syntax for causation.” (This is a cousin of the argument I made in A semantic layer is not a context layer: consistent definitions tell an agent what the data means, not how the business works.) A good analyst reasons from a model of the business in front of them, not from table names. An agent will too, if someone has written the model down.

And the discipline I applied by hand should not have been mine to apply. Anthropic publish a redacted version of their actual warehouse skill file, and it mandates a footer on every answer: source, confidence, freshness, owner. That is my Gong spreadsheet, except that it arrives whether or not the person asking knew to ask for it. Written once into the skill file, it applies to every answer the agent gives, including the answers given to people who would never have thought to check.

I should declare an interest. Making sure the right data exists, well created and carrying its meaning, is what we do at Snowplow, so of course I would say all this. Read it with that in mind. Every number above is Anthropic’s.

Problem two: the wrong question, answered perfectly, and I do not know how to fix it

Everything above solves the first problem. It does very little for the second.

A good analyst never takes a question verbatim. They push on it, work out what is really being asked, and name the assumptions inside it before running anything. Most bad analysis I have seen was competently executed; it answered a question nobody should have asked. In twenty years I have not found a reliable way to teach that skill, and I do not think the profession has either. It is the hardest thing an analyst does and the thing we have never known how to hand on.

An agent that answers precisely what was typed skips that step entirely. The best partial mitigation I know is to make the agent say which question it decided to answer and what it assumed to get there, so that a mis-framing is at least visible on the page. Anthropic state the problem exactly, without offering a remedy: an agent that does not understand your business “will answer what the user asked, but not what they meant.”

But I want to be honest that this is a much weaker answer than the one I gave for the first problem. Declaring assumptions helps the person who already suspects the question might be wrong. It does nothing for the person who does not, which is most people, most of the time. For the first problem Anthropic can tell me it is tractable and roughly what it costs: 21% to 95%, and here is the work. I have no equivalent for the second. I do not know how you build a system that reliably challenges the question it has been handed, and I have not seen anyone who does.

The one direction I would look, and this is a hunch rather than a plan, is at the work being done to make these models good teachers. A good teacher does not answer your question either. They ask you what you are actually trying to find out, and they are often right that it is not what you said. If that behavior can be trained for tutoring, it may be trainable here. I want to think about it properly rather than gesture at it, so I will come back to it.

That is not a satisfying place to leave it, and I am leaving it there because I would rather say so than pretend.

What is left for culture is much smaller, and it is real

None of this makes the culture argument go away, and I am not smuggling in a claim that good data is sufficient. It is not. Well-created data removes one set of obstacles. It cannot make anyone care whether the answer is true.

But what I would now call data culture is a much narrower thing than what we spent twenty years asking for, and it is not literacy. It is not teaching people to write SQL, or to read a confidence interval, or to know when a median beats a mean. It is one norm: that a number gets asked about. Concretely, two questions being ordinary to ask out loud in a meeting. How did you get that? And is that the question we should be asking?

That is what made me check the 57%. Not skepticism in the abstract. I expected to be asked the first of those questions, so I made sure I could answer it. Nobody acquires that on a course. People acquire it when asking is normal and being unable to answer is uncomfortable.

It is also, as far as I can tell, the only thing that touches my second problem at all. I cannot build a system that reliably challenges the question it has been handed. A colleague can. Making it normal for someone in the room to say “hang on, why are we asking that?” is not a technology fix, and I am not going to pretend it is a complete one, but it is the part of the old culture argument that survives contact with all of this, and it is the part I would spend real effort on.

It is also a much smaller ask than the one we used to make. We are not asking anyone to learn to get the number themselves any more. We are asking them to expect to be asked, and to want the answer to be right even when it is inconvenient. Neither of those needs a curriculum or a certification, which is what twenty years of data literacy programs tried to give people.

A test you can run on your own roadmap

So here is a test you can run on your own roadmap this quarter. Take the list of everything you are planning to do about the risk that your stakeholders use AI badly. Count how many items require your stakeholders to change, and how many require you to build something. If the first number is bigger than the second, you are running a data literacy program with the word data crossed out. We ran that program for twenty years. Adoption sat at 25% for seven of them.

We got this wrong for twenty years and the software rescued us anyway. It will not rescue us twice, because this time the failure is silent. A dashboard nobody opened was a visible failure; you could count the logins and know the program was not working. A confident, wrong number in a board pack is not visible at all. Nobody will report that adoption has stalled. The answers will keep arriving, fluent and fast, and they will be wrong in ways that only show up much later, in the decisions.

Blaming the user has always been available to us. It never requires building anything, it survives contact with a budget, and it is the reason the last twenty years went the way they did. It is on the table again right now, dressed up as AI literacy, and it is going to be very tempting.

So I will ask something narrower than I asked last time. The first of my two problems has an answer and the answer is work: write down what your business means, and make the system do the checking rather than the person. The second problem I cannot see my way through. If you have found a way to make a machine push back on a question instead of answering it, I would like to hear about it. I would rather we worked that one out in the open than each of us discovered separately, a quarter too late, that we have been answering the wrong question very well.

I have been lucky to spend a lot of my time with the data leaders at our customers and prospects. One of the things I consistently try to understand is what their stakeholders want from them, what they are trying to achieve themselves and how much of their attention goes to improving the business versus pursuing technical goals.

Two things have surprised me in the last six months.

The first is that every data leader I have spoken to reports the same thing: a marked increase in business stakeholders using the data their team produces, across a wider set of use cases. All of them attribute it to AI. They report that this is not a gradual improvement on a long-running trend: it is a step change.

The second is the real surprise to me. In my experience up until now, widespread adoption by the business of data has only happened in organizations where:

1. The data leader has obsessed over driving that adoption

2. The CEO and leadership team are themselves very data-driven and very publicly set a strong example for the rest of the organization

3. The data leader spends nearly all of her time focused on driving that adoption and delegates responsibility for designing, building and managing the data platform to her team. In practice this means she is mostly with the business and has a crystal clear understanding of what the biggest challenges are for the business and has driven clarity across the org in how data can support solving those challenges

This is the first time I have seen organizations enjoy widespread data adoption even though the above is not true of those organizations.

AI has enabled organizations to go data-driven that previously could not

There is a failure mode in data leadership that every data leader will recognize, because most of us have done it. We spent our time architecting an excellent data platform and not nearly enough time going deep with our business stakeholders, making sure that what we built actually helped them solve their problems. The engineering was the part we were good at, it had clear success criteria, and it did not require us sitting through somebody else’s planning meeting. Sometimes the stakeholders came along and used what we had built to solve their problems. But often they did not, and the lesson everyone drew was that building it was not sufficient: the teams that made a real difference were the ones that spent their time doing the harder, less glamorous work of driving adoption.

That work was hard. Worse, it felt never-ending. Even the data leaders I think of as best at it complained about the effort of getting data properly baked into how decisions were made. They would win a team over, embed an analyst, get the metric into the weekly review, and then someone would be promoted and they would start again. The work never stopped.

Almost everyone attributed that to data culture and data literacy. It was the standing complaint at every industry event I attended for a decade, and it was in every report. Qlik and Accenture commissioned a survey of 9,000 workers and found 21% fully confident in their data literacy and 74% “overwhelmed or unhappy working with data.” Thomas Redman, in Harvard Business Review, described the mechanism as self-reinforcing: “Lack of talent breeds fear, exacerbating difficulties in adopting a data-driven culture.” That is exactly what it felt like from the inside. Every other function would call that a product problem. We called it a customer problem.

So you would expect this step change to appear first and most strongly where data teams had done the commercial work, and weakly or not at all where they had not.

That is not what is happening. It is showing up everywhere, in the same way, at the same time.

This is great! But it warrants examination

Before anything else, this deserves celebrating! We have been trying for at least two decades to democratize data and enable our organizations to make data-informed decisions, and now it is finally happening at scale.

More people in more businesses are using data to make more decisions than at any point in my career. Not looking at a dashboard someone else built: asking their own questions, of their own data, about their own problems, getting answers, reflecting and then iterating their thinking. It is also happening to the people who were most excluded before. The marketing manager who never learned SQL. The ops lead who filed a ticket and waited three days. The finance analyst who exported to Excel because it was the only tool that did what she needed. They are all in the data now.

I do not want to be the person who receives that news and immediately explains what is wrong with it.

But look at how strange this is. Something that was supposed to be a culture and literacy problem has been substantially moved by a technology breakthrough: not addressed at the margins, but moved, in six months, in companies where nobody ran a new training program.

If a change in tooling can shift a number that twenty years of culture work could not, then either the diagnosis was wrong, or it was right and something very unlikely just happened. I think the diagnosis was wrong. And I held it.

The interesting question is not what we got wrong but how. That is what this piece is about. It matters because the same mistake is on the table again, with AI, which is where I will pick this up in the next post.

What did we get wrong?

We believed that the barrier to a business making real use of its data was culture, people and literacy, not technology. It wasn’t that “culture also matters.” We believed that technology had essentially stopped being the constraint, and it was people and processes that were holding back data adoption.

It was also well evidenced. In a study fielded at the end of 2021 and published in early 2022, BARC and Eckerson Group surveyed 214 data and analytics leaders: “The percentage of employees actively using BI/analytics tools is currently 25% on average, reflecting minimal growth in the past seven years we’ve been tracking this metric.” Wayne Eckerson, in the same release: adoption “has been stuck around 20% for many years.” Seven years of tracking, no movement, after two decades of investment in self-service business intelligence. (Usage of BI output did rise over that period. The plateau is in the number of people asking their own questions, which is what is at issue here.)

The earliest good statement of the diagnosis I can find is from 2010: MIT Sloan Management Review and IBM’s Institute for Business Value surveyed around 3,000 executives across 108 countries: “the adoption barriers organizations face most are related to management and culture rather than being related to data and technology.”

Nine years later, three McKinsey partners writing in Harvard Business Review compressed it into seven words as their standfirst: “Technology isn’t the biggest challenge. Culture is.”

The load-bearing evidence came from one annual survey: the NewVantage Partners executive benchmark, run by Randy Bean and Tom Davenport, which asks the most senior data leaders at around a hundred large companies to name the principal challenge in becoming data-driven. Bean reported the 2020 results himself: “Technology does not appear to be a barrier or the problem. Only 9.1% of executives pointed to technology as the principal challenge to becoming data-driven.” Culture, people and process: 90%.

Most of the numbers in this argument trace back to that survey, including many that get attributed to MIT or McKinsey because of where Bean published them. It’s worth describing the study: around a hundred respondents, one per company, invitation-only, heavily weighted to financial services, with “culture” a composite bucket covering people, process and organizational alignment. It records what data leaders believe about their own programs. It reflects their thinking rather than a rigorous analysis of what was actually holding back adoption.

I believed all of this. I said versions of it on stages and in sales meetings, while selling data infrastructure. So did nearly everyone I respected in the industry: as late as December 2021, less than a year before any of this began to change, the claim that technology was not the barrier, and that culture was, still read as settled common sense.

Even today data leaders are still saying culture is holding back adoption of data

Even while AI is driving adoption of data in organizations, old claims about culture holding this back remain prevalent.

The 2026 edition of that same survey, published last December, records the highest culture reading in the survey’s history. From Davenport and Bean’s foreword:

“One of the flaws that we’ve seen year after year is that the primary obstacles to effective use of data and AI are almost entirely human rather than technological. This year the highest-ever percentage in this survey (93%) pointed to human issues — culture and change management — as the key challenge to data and AI adoption. Only 7% blamed technology. But how many CDOs spend 93% of their own efforts addressing human issues?”

Ninety-three percent, and a closing question that asks data leaders to put more of their effort into the human side. Published into a year in which a technology change is visibly moving the number that twenty years of human-side effort could not.

If technology can fix it, adoption was never a culture problem

If culture prevented adoption of data, a change in technology could not have produced a step change in adoption, because there was almost nothing left for technology to fix. Something else would have had to change: literacy, sponsorship, organizational design, incentives.

Nothing else about data changed. Nobody ran a literacy program in 2025 that finally worked. And the step change is uniform across organizations with very different levels of data culture and literacy, which is fatal to the strong version of the diagnosis: if culture were the binding constraint, results would vary a lot more across organizations. My experience is that they do not.

The obvious objection is that AI was itself a culture event. CEOs mandated it, boards asked about it and people met it at home before they met it at work, which is exactly the sponsorship and personal familiarity that twenty years of literacy programs were trying to manufacture. That is partly true, and it is the strongest counter to what I am claiming. But look at what it concedes. The sponsorship arrived for AI, not for data. No executive stood up in 2025 and told their organization to care more about the warehouse. What the mandate did was hand people a different door into the same data, and they walked through it. If culture only moves when you ship people a better interface, the interface was doing the work all along.

So technology was a binding constraint and a large one, and we spent twenty years telling each other it was not. That does not show culture never mattered. I think it matters as much as it ever did, and I will come back to it in the next post. It shows that the confident, near-unanimous claim that technology had stopped being the constraint was wrong. We had a variable that our own benchmark put as low as 7%, and it turned out to be the one that moved everything.

Before going further I should own something, because the irony has not escaped me. My evidence for the step change is the reported beliefs of data leaders, gathered in conversations rather than surveys, and that is exactly the class of evidence I have just been asking you to distrust. The independent numbers do not yet show it: Gallup surveyed 23,717 US employees in February and found only 18% of frequent AI users doing data or analytics work with it, and the Census Bureau has the share of businesses using AI flat since last December. So the claim I am making is deliberately narrower than “everyone is self-serving now.” It is that for the first time in twenty years the interface constraint has visibly moved, and that where it has moved, in the companies that had already done the unglamorous infrastructure work, it has moved a long way. If you are seeing something different, I want to know: that is the question this piece ends on.

What actually drove data adoption was a new interface on the data infrastructure that was already there

The technology that changed was not the data platform. It was the interface to it.

A change of interface was sufficient because in most of the companies I am lucky enough to work with the hard infrastructure work had already been done, for reasons that had nothing to do with AI, long before anyone knew that an agent would be both the interface to data and a consumer of it. Data consolidated in a warehouse, modeled into something coherent, definitions agreed and written down somewhere. That architecture took a hammering in 2022 as over-tooled and over-priced, and the cost critique was fair, but it was mostly built, and it turned out to be critical in enabling the business to use AI to think in a data-driven way.

The demand for data was latent all along. The business was never allergic to data. It was allergic to BI tools, to SQL, to filing a ticket with the data team and to waiting three days for an answer it needed on Monday. Take the tax away and the appetite is there.

The most damning detail is in that same BARC survey. Asked what was driving increased usage of their tools, only 14% of those leaders named natural language query. This was a year before ChatGPT. And the barriers they named, in their own order: lack of training 50%, poor data quality 41%, budget 36%, ease of use 33%.

Training first. Ease of use fourth.

Some people saw it coming. Tristan Handy, writing about the modern data stack in late 2020, two years before any of this landed: “There is no technical hurdle to be solved here—the building blocks are all in place. The hard part is figuring out the UX.” And Benn Stancil went at the culture diagnosis by name in August 2021: “data cultures don’t materialize out of employee handbooks or internal seminars. The structures of our technologies and organizations till the land from which they grow.”

We were the only function that concluded its customers were the problem

For twenty years the data profession’s settled explanation for its own limited impact was that the people it served were not good enough at receiving what it produced. Not literate enough. Not curious enough. Not sufficiently sponsored from above. Our answer was to send them on courses.

Those people were our customers. Every data leader I know describes their stakeholders in those terms and means it sincerely. And no product team on earth would survive concluding, for two decades, that its users simply needed more training. They would have been told, correctly, that a product requiring a literacy program has a design problem.

We told ourselves the opposite. Forrester had described the goal, in 2010, as self-service BI “where the BI developers and business analysts are one and the same person.” Turn the business user into an analyst. That was the cure as we understood it. It reads now like a description of the disease.

I do not think the last twenty years were wasted. The warehouse, the models, the definitions argued over and written down: that work is precisely what made the new interface useful, and the companies where the step change is biggest are the ones that had already done it. What was wasted was the diagnosis. We spent two decades studying our stakeholders when we should have been studying the front door.

So I would be interested to know whether others are seeing the same step change I am, or something different. My evidence is a set of conversations rather than a survey, which, as I have already admitted, is exactly the class of evidence I have spent this piece asking you to distrust.

If I am right, I worry that we are about to make the same mistake again.

In the second part of this piece I look at where the gap between data people and everyone else has moved to, why “teach people to be better with AI” is the same flawed approach we took with data for twenty years and what we need to build instead.

A semantic layer is not a context layer. It does a real job, and a durable one, but it is a fraction of the job many agents acting in the enterprise actually needs done. And the rest of that job is not a matter of putting more data into the semantic layer. This piece is about the gap: what a context layer has to do that a semantic layer does not, and why anyone putting agents into production should care which one they are actually building.

I should declare my interest. At Snowplow we are building a customer context layer, for agents that work with customers, whether directly in an application or by supporting the marketing and support agents that serve them. Read what follows with that in mind. The argument stands or falls on its own logic.

Context layer: the phrase everyone is using but no one has defined

Start with what is not contested, because it shows how precise this language can be. “Context window” is exact: Anthropic’s documentation calls it the model’s “working memory,” all the text a model can reference when generating a response. “Context engineering” converged fast: Tobi Lütke reached for the term in June 2025, Andrej Karpathy sharpened it a week later into “the delicate art and science of filling the context window with just the right information for the next step,” and LangChain and Anthropic landed in the same place within months.

Then you reach “context layer,” and the ground gives way. Enterprise Knowledge, a consultancy with nothing to sell in this particular fight, is blunt: practitioners “are still struggling to agree on the terms themselves.” Different camps use the phrase to mean materially different things: governed metadata over the warehouse (Atlan, and the reading analysts applied to Snowflake’s Horizon Catalog at Summit), agentic retrieval across both structured and unstructured information (Contextual AI), a knowledge graph (Enterprise Knowledge again), or the live, identity-resolved state of a specific entity, the camp Snowplow and Amperity are in.

But underneath, every camp makes the same move: each assumes the job is to give an agent access to data that already exists, differing only on which data and in what shape. That assumption fails twice. First, access is not the hard part: context has a real cost, so it is not enough to make data reachable; it has to be the right data, optimised for this agent at this moment. Second, it counts only the data the enterprise has already captured, and says nothing about the data that has to be created before any of it can run.

Give the semantic layer its due

To see the difference clearly, give the semantic layer its due, because it is a real and durable idea that long predates this conversation.

It was, in effect, invented by Business Objects. Their patent, filed in November 1991, describes a system that lets “end users access (query) relational databases without knowing the relational structure or the structured query language,” wrapping the metadata model in a container they called a “universe.” The label came later; the thing is thirty-five years old, and its job has never changed: the consistent interpretation of data that already exists. What does “revenue” mean? It is the agreed dictionary that stops every team in the building defining “churn” differently.

The self-service-BI wave ushered in by Tableau largely did away with the semantic layer: analysts pointed a visualisation tool straight at the data and each re-implemented their own definitions. Looker brought it back in 2012 with LookML, so many people could work consistently with the data the cloud warehouses were making available. It is having a full renaissance now, with dbt, Cube, AtScale, Snowflake’s Semantic Views and Databricks’ Metric Views, for a reason that has nothing to do with humans: an agent that guesses what “revenue” means is worse than useless. The clearest tell that the term is live and contested: in September 2025, Snowflake, Salesforce, dbt Labs and others launched a standard, the Open Semantic Interchange, to make their semantic layers interoperate. You do not standardise something everyone already agrees on.

So the semantic layer is enjoying a deserved second life. The mistake is not in valuing it. It is in thinking that makes it a context layer.

The difference is the job, not the data

The tempting way to distinguish the two is by their contents: to say a context layer just has more in it, usually the documents and transcripts that never fitted neatly into tables. But that invites an obvious rebuttal: then put more data in the semantic layer. If the only difference is what is inside, the distinction collapses, because in principle you can put anything inside anything.

The distinction is not the contents. It is the job. A semantic layer answers one question: what does this data mean? It is definitional, retrospective, aggregate, and the same dictionary whoever opens it. A context layer answers a different question: what does this agent need to know to act well, here, now? Put in one line: a semantic layer describes the past consistently; a context layer assembles the present for a decision. Two things separate them. One is tense: the settled past versus the live present. The other is address: a semantic layer is unaddressed, indifferent to who is asking or why; a context layer is written for one reader, this agent, in this situation, trying to do this thing. Tense and address together, and everything else in this piece follows from them.

The word “context” is overloaded, so let me fix the vocabulary. I will use situation for the raw thing an agent is dropped into: the environment, the entities in it, and what each is trying to do. And I will reserve context for the refined, decision-ready package assembled both from that situation and for it: from it, to understand what is going on at all; for it, when handed back to the agent so it can take its best next step. The situation is the raw material; context is what you manufacture from it. A semantic layer is one ingredient in that manufacture. It is not the factory.

So what is the job, exactly?

The job of a context layer is to give an agent the most relevant, precise, succinct yet complete picture of the situation it is in, together with the knowledge it needs to act, so it has the best possible chance of success, however the agent defines success. (”Succinct yet complete” is deliberately a tension: enough to act on and no more, because context is not free.)

The interesting question is how, and it breaks into a five-step pipeline that runs not once but over and over, learning as it goes.

1. Comprehend the situation. Build an accurate picture of the environment: which entities are involved, their current state, what each is trying to do. Get this wrong and everything downstream is precise about the wrong thing.

2. Frame the problem. Translate the situation into the specific problem to solve. One situation affords many; this picks the real one. Get this wrong and you solve the wrong problem flawlessly.

3. Determine what is needed. Map that problem to the information and knowledge required to solve it. Get this wrong and you retrieve a pile of plausibly-related, useless material, or miss the one input that mattered.

4. Acquire it, across every source. Gather it wherever it lives: tables, knowledge graphs, documents, real-time event streams, external systems. This is the only step a semantic layer touches, and even then only the slice whose answers sit in governed warehouse tables.

5. Compose it for use. Arrange the material into the form most usable for this decision by this consumer: the right granularity and precision, disambiguated, no more than is needed. Get this wrong and you hand the agent a technically-complete data dump it cannot use.

The semantic layer contributes to step four, for one source type, and is silent on the rest. That is not a criticism; it is simply a much smaller part of the job than the conflation implies.

This answers the obvious objection: can’t I just put an agent on top of my semantic layer and let it do the rest? You can, and for one case it is the right architecture. Picture an analytics agent, an internal user asking the warehouse questions in natural language: “What was net revenue retention in EMEA last quarter, and which segments dragged it down?” The agent handles steps one to three by reasoning, the semantic layer handles step four, and the answer is good, because the situation is simple, the data already exists in the warehouse, and it is already aggregate. From inside that case, “semantic layer equals context layer” looks obviously true. But notice what that arrangement is. The agent supplies the reasoning, which a capable model can do for itself. What it cannot do for itself is the infrastructure: making sure the right data was created in the first place, acquiring it from sources the semantic layer does not reach, and composing and delivering it in time. Split the job that way and the boundary draws itself. The context layer is the infrastructure half, and it is precisely the half a semantic layer does not provide.

It all rests on data that has to be created

The most important thing the decomposition surfaces hides inside steps one and four: both presuppose that the relevant data exists. Step four quietly assumes the thing you are reaching for was recorded somewhere. Often it was not. Step one is more demanding still: you cannot comprehend a situation that was never captured. Describing a situation accurately is not a passive act of reading; it is an act of creating the data that represents it.

Take two agents facing very different decisions. An incident-response agent has to decide whether to roll back a release the moment a service starts erroring: what settles it is which deploy shipped in the last ten minutes and which downstream services are degrading right now, not last quarter’s uptime. A customer-facing agent, handling a live chat or working inside an application, has to decide what to say or do next: what settles that is what this customer just did, that they hesitated on the pricing page, abandoned checkout, and came back twice in the last five minutes. In both cases the context that matters is made of events happening now, and it exists only if those events were deliberately captured as they happened. Typically, neither situation sits in a table waiting to be queried.

So the job of a context layer is not, fundamentally, to support a retrieval process. It is to ensure the data needed to run all five steps has been designed, created and made available in the first place. A semantic layer assumes that work is done. For a real-time agent it usually is not, and no amount of consistent definition conjures data that was never recorded. (This is the thread back to where this series started: a context layer is where created context gets served; data creation is how that context comes to exist.)

This is general, and worth proving general by stepping outside the customer-data world I happen to live in. Take a coding agent. It needs the current state of the codebase, the diff in flight, the failing test, the ticket, the team’s conventions, and, above all, why a piece of code was written the way it was. That last one is a decision trace, and decision traces are almost never recorded: they live in someone’s head, or a Slack thread that scrolled away. A semantic layer over a metrics warehouse is irrelevant here; this agent’s context is a code graph, live editor state, and institutional knowledge that has to be captured to exist at all. Or take a clinical decision-support agent at the bedside: live vitals streaming now, a history resolved across record systems that share no identifier, medical knowledge that is graph-shaped and constantly revised. A consistent definition of “readmission rate” is no use mid-ward-round.

Three different domains, one structural fact in each: much of what the agent needs is graph-shaped rather than tabular, real-time rather than batch, and often not recorded until something deliberately records it. None of those three properties is what a semantic layer is built for.

And it has to arrive in time

The decomposition makes a second thing visible, the one the warehouse path most often founders on: composing the perfect context and delivering it a moment too late is a total failure, and a different failure from getting the content wrong. Semantic layers are typically built on the gold tables at the end of a batch pipeline. That is the right place for consistent reporting and the wrong place for a decision happening in a live conversation. The agent is mid-sentence with a customer, or mid-edit in a file. Context that is correct but arrives with last night’s batch is not context; it is history. This is the tense difference made physical: the past, however consistently described, arrives too late for the present.

The loop that makes it better

One more thing, and the part I am least willing to draw a clean box around. A good context layer gets better over time: it records what the agent did and what happened, and feeds that back in, which is itself an act of creating new data, the decision traces step three will want next time. That loop is shared, and cannot be optimised from one end. The agent owns the action and its outcome; the context layer owns turning that outcome into durable, retrievable signal. Optimise one without the other and it breaks: an agent that learns nothing it can pass on, or a layer of traces no agent ever acts on.

Why the distinction is worth defending

The failure mode is by now easy to picture: a flawless semantic layer, every definition agreed, that can still only tell a real-time agent what the terms mean and what this segment did last quarter, when what it needs is what is happening right now. That gap is the space a context layer occupies, and it is why the definition currently forming, context layer as semantic-layer-plus-agent, is worth resisting: it is a definition the data platforms already meet by construction. If that version wins, the category was settled the moment it was named.

I want to be fair to the most serious version of the opposing view, because serious people hold it. Snowflake’s own engineering team has written one of the sharpest accounts of what agents need from data. Their “Agent Context Layer” post lays out six stacked layers and makes a good prediction, that “winning architectures will treat agent context, and not the model, as the core product, as the model gets commoditised.” I agree with that sentence, and a warehouse can assemble a great deal of context. The disagreement is narrower and more structural: the real-time, entity-resolved, behavioural creation at the front of the five steps is the part the warehouse-native version cannot do, not through any failure of engineering, but because that data was never in the warehouse to begin with. A semantic layer is a tool for interpreting data, not for bringing it into existence.

So: a context layer is defined by a job a semantic layer does only a sliver of, and that job rests on data that has to be created, resolved to a live entity, served in time, and improved through use. Get that general definition right and any more specific claim, a “customer context layer,” say, follows from it rather than having to be asserted.

A closing note on how to do this honestly. The temptation in pieces like this is to invent a term and plant a flag on it, and coining a word when adequate ones already exist is the surest sign someone is trying to own a category rather than describe one. So I am not going to coin anything. I would rather build inside the vocabulary that already exists, Brinker’s “systems of context”, Karpathy’s “filling the window,” define the job as precisely as I can, and let you judge whether what anyone builds actually meets it. That includes us: if “customer context layer” earns the name, it will be because it does the job, not because we said it first. A clear definition is the one thing the people best placed to own a term have the least incentive to supply. I would rather supply it, and compete on whether we meet it.

.

That is the conviction this whole Substack will start from: agents must not run on data exhaust — and far too many of them do. We were told for years that data is the new oil, something that pools beneath the organisation waiting to be pumped and refined. It was a poor metaphor then; in the age of agents it is a dangerous one, because it tells you the valuable data is already there for the taking. It almost never is.

It has been a long time since I last wrote here, and AI has reshaped the data landscape in the gap — exhilarating, and in equal measure hard to keep up with, let alone make sense of. This piece is my way back in: why, in a world increasingly drowning in machine-made data, deliberately creating data worth trusting matters more than it ever has. Let’s start with how much the word “data” now has to carry.

What we now call data

The category “data” has exploded.

For most of the history of our industry, “data” meant the structured, tabular numbers that fit in rows and columns, typed and schema’d, the kind of thing you could put in front of a BI tool. Text, images, audio and video were “data” only in the loosest sense. You could store them (in an object store, not a database), but you could not do much with them.

That boundary has gone. “Data” now encompasses text, images, audio and video; it can also refer to agent outputs, tool calls, reasoning traces, application code, webcam footage and the synthetic data the models generate themselves. The deeper shift is in where the boundary sits. Data used to be bounded by what fits in a table, because that was a pre-requisite to derive insight from it. (That was true whether you were doing analytics or most flavours of machine learning.) With the birth of language models, that is no longer true: it is now bounded by what a large language model can reason over, which today is pretty much everything we can represent digitally.

This is thrilling: it opens up whole new problem types that computers simply could not touch before.

How you impose order on that data sprawl is a really hard question. I am going to leave it hanging for now, because it is the subject of the next piece. The more urgent question, for now, is who, or what, is now consuming all of this?

Agents: a strange new data consumer

For most of the history of our industry there were only ever four ways to drive value from data, and every one of them ran on the same thing: a table. Descriptive: aggregate the rows to work out what happened — sum a column, count the records, group by a dimension. Diagnostic: dig into why, slicing that same table by dimension and comparing a metric across cohorts, or reading the correlations between its columns. Predictive: lay the data out as a feature matrix — one row per example, one column per signal, plus a column for the thing you are trying to predict — and fit a model to it. Prescriptive: hand the decision to the data itself, a pricing model or an approval engine that will not fire until it is handed a row of typed, expected inputs. Four levels of ambition, one substrate beneath all of them: always rows and columns.

Now hand an agent data in the broad sense, and what it can do looks nothing like that list: summarise a thousand documents, riff on a theme or brainstorm against one, build an argument or take one apart, propose a hypothesis worth testing, decompose a problem into constituent parts, turn a messy situation into a diagram or a video. None of this is analytics. It is a different kind of work entirely, and the material it runs on is no longer confined to rows and columns.

In the old world, structure was the price of admission. Dashboards, recommendation engines, decisioning apps: rigid programs that would not start without typed, schema’d inputs. The one apparent exception that proves the rule: a narrow class of machine learning - e.g. sentiment analysis - ingested raw text. But it delivered a structured output, a score or a label, and slotted straight back into the structured world. Unstructured data was never reasoned over in place. It was something you converted into structure as fast as possible, so the real tools could use it.

Meet the large language model, then, and meet it for now through a single face: as a data consumer. It is the first widely deployed thing that consumes unstructured data as unstructured data, reasoning over it directly with no conversion-to-structure step in between. Its appetite, the context window, is finite, a hard token budget, but it is structure-agnostic. It will take almost anything, with none of the schema constraints we built the entire discipline around.

It is tempting to read that as liberation. Structure is dead; just hand the model the pile. It is not. Removing the requirement for structure does not remove its value. The context window does not care whether you hand it a pristine star schema or a heap of raw transcripts, and that is exactly the problem. The model will accept anything, but it reasons well only on faithful, well-organised context. Structure stops being a technical requirement and becomes a quality choice.

The machine that is also the material

Now I can put back the two faces of the model I asked you earlier to ignore.

Let’s categorise everything that touches data by what it does to the data. Some things create it, producing data that represents the world in some way: trackers, sensors, transactions. Some process it, turning data into more data: a SQL query, a dbt model, a training run. And some consume it, turning data into a decision, an action or an insight. These three roles were always kept apart, each played by its own kind of component in a production line. The large language model is the first technology that can play all three roles:

• it is a source of data, generating text, code, decisions and traces, and driving an explosion in how much data the world produces;

• it is a processing step, a station on the line that transforms its inputs into something new; and

• it is a consumer, reasoning over the context window to reach a decision and take an action.

This is unprecedented on two counts. We have never had a single component that is at once the raw-material supplier, the machine and the customer. And it occupies all three roles having shed the structure requirement that defined every one of them before. A line that used to run in one direction, through distinct stations, collapses into a single structure-agnostic component, and that bends the manufacturing line into a loop. The machine is now also the material.

This is the idea the rest of this Substack will keep returning to. Everything that follows, the promise and the danger both, comes from those three roles collapsing into one.

Choking on our own exhaust

The risk falls straight out of the loop, which is why it needs no separate motivation. If the line loops, and we keep feeding it the exhaust it throws off, quality does not merely stall. It compounds downward. The assembly line starts eating its own output.

This is not a thought experiment. It is model collapse: the finding, published in Nature, that models trained recursively on their own generated data degrade across generations until their output turns to nonsense. It is context rot: the documented result that frontier models get worse as you cram more into the context window, not better. It is agents grounded on the undifferentiated output of other agents. At machine scale, “don’t build systems out of a byproduct” stops being good hygiene and becomes something closer to existential.

A caveat, because this is exactly where the careful reader objects: none of this is an argument against synthetic data. Model collapse is what happens when a model is trained recursively on its own output with no fresh, real signal coming in — not whenever a machine has a hand in making the data. Synthetic data created on purpose, to fill a known gap or stress a known edge case, is some of the most useful data there is. The line that matters is not human-versus-machine origin. It is whether the data was made for a purpose, by design, or scavenged as the undifferentiated byproduct of something else.

But the same loop runs the other way, and this is the hopeful part. Feed the loop faithful data, made to spec, and quality compounds upward. That is what a learning system actually is: manufacture a decision, deliberately create data about it (what was decided, how, and the outcome it drove), reflect on whether that outcome was the one you wanted, and feed that back so the next decision is better. Over time, reliably, it improves.

Collapse and learning are the same loop. The only difference is whether the data flowing back was created or scavenged. That single choice decides whether quality compounds up or down, and it is more or less the whole argument in one sentence.

The lovely example of data that we have identified agents need, and need to be manufactured to spec, is the decision trace — a phrase I am borrowing from Jaya Gupta and Ashu Garg of Foundation Capital, who named it late last year. A decision trace is a record of how a decision was actually made: what was weighed, what was discarded, why one path was taken over another, who signed it off. Almost nothing emits this: it needs to be deliberately created. It survives, if at all, as fragments buried in call transcripts and meeting minutes, never with the granularity, fidelity or provenance an agent would need to learn from it. And it is exactly what an agent that makes decisions needs: a faithful account of how humans reason through a choice, to follow or to disagree with.

Notice what that means. Even now, with machines generating more data than we know what to do with, some of the most valuable datasets still do not exist. They have to be manufactured on purpose, because no system throws them off as a byproduct.

When data is effectively infinite and machines make most of it, the scarce thing is no longer data. It is data that is actually useful. The data that is worth trusting. The abundance of exhaust does not make deliberate creation less valuable; it makes it close to the only thing that matters.

What I want to write about

That is why I am writing again: to think in public about how we create valuable data worth trusting, in a world where that has never been easier to fake or more expensive to get wrong. Data Creation - the thing this Substack is named for - is just that: the deliberate act of making data for a purpose, not the scavenging of data made for some other one. Some of it will be conceptual. A lot of it will be the unglamorous, specific things we are learning from doing the work. (Snowplow is where most of that learning comes from, so it will show up here, as evidence rather than a sales pitch.)

The next piece picks up exactly where this one stops: with the layer the whole industry is currently arguing about, and the one term it keeps getting confused with. Everyone has started talking about a “context layer.” Almost no one agrees what it is, and a good many people are quietly assuming it is just a semantic layer with an agent bolted on top. It is not, and the difference, it turns out, matters enormously.

More soon.

In this post, I’m going to look at a separate but very related trend: traditionally, companies have driven most value from their data in reports and dashboards. That is changing rapidly - today companies are increasingly building their own data applications to drive value from data. But what exactly is a data application? What is driving this shift to data applications? How should companies get started building data applications? In this article, I answer these questions, and also explore why data products are powerful tools to support organizations building data applications.

Image courtesy of Dall-E

What is a data application?

Ten years ago, organizations primarily used data to power dashboards and reports (e.g. business intelligence).

But with the rise of AI, digital platforms and improved connectivity, more and more companies are building data-intensive applications: applications that use data to power significant parts of their business. 

Many of these applications are operational systems - they form part of the technology estate that companies use to deliver their services to their customers. There are many examples of data applications, to list just a few:

Real-time classifiers

These are real-time AI applications that make a decision, for example:

• Fraud detection: algorithms that decide if this individual is trying to commit a fraud, in which case they block a transaction from completing

• Bot detection: algorithms that decide if a visitor is a bot or a human, and potentially take an appropriate action. (This may include blocking a bot further access to a website, or simply labeling and filtering the data associated with the bot so it does not contaminate KPIs like ad views.)

Recommendation and personalization 

It is relatively common for algorithms to power parts of an in-website or in-app experience, for example:

• “People who bought this also bought this”

• “Other products you might be interested in” (these might be served on a side bar along the checkout to encourage customers to buy additional items, for example)

• Search results can be personalized based on preferences that an individual reveals but doesn’t explicitly state in a search (e.g. AutoTrader example).

In this case, the AI selects the “best” content or products from a long list and returns them to the user. 

Decision-support systems

The above examples are  data applications that make a decision (e.g. decide that this user is committing a fraud or that these specific products are likely to be of interest to this visitor). In addition, applications can also be built that use data + AI to help individuals (or group of people) make a decision. 

A nice example is an apparel retailer (and Snowplow customer) that has developed a data application for its buyers, that:

1. Ingests the list of the SKUs they might want to stock for the next season. Note that in many cases, these SKUs are just rough sketches or concepts of items that could exist but are yet to be developed

2. Provides a set of recommendations as to which products should be purchased for the forthcoming season, including:

   1. How much of that product to buy

   2. The rationale for its decision. (E.g. “This product sold particularly well last season and is in a product class that has seen steady sales growth over the last three years. Further, last year’s profit was lower than it could have been because not enough of the product was purchased to meet demand”)

3. Enables users to select a particular item and then show the forecast demand data alongside historical availability, size curves and similar products

The application is then used to help the buying team decide what to purchase for the next season. The buying team makes the actual buying decisions, but these decisions are no longer just based on gut. Instead, they’re based on a thorough understanding of the past performance of similar items and the impact of seasonality.

A totally different example from the media industry has some surprising similarities. 

A newspaper group (also a Snowplow customer) built an application for its editors to help them optimize the placement of stories on the homepage through the newscycle. 

The application showed the homepage as it was, but annotated it with data that showed the performance of individual content items in each place, benchmarked against the previous performance of content in that same item location. (Because placement is such a significant driver of content performance.) 

The data made it easy for editors to recognize content that was over or underperforming (for example overperforming if the story was getting more popular, underperforming if it was losing relevance), which made it much easier for the editors to confidently rearrange the homepage frequently over the next cycle.  

Like the buyers at the apparel retailer, the editors at the newspaper group had an application that was built to meet their needs and surfaced the data in the most helpful way to empower them to make a very specific but critical set of decisions on an ongoing basis. 

Dashboards and reports

Old fashioned dashboards and reports can also be considered data applications. Many dashboards are specifically designed to help users make certain decisions (e.g. optimizing the deployment of money across different campaigns by highlighting campaigns that are over or underperforming in terms of return on investment) or to monitor the health of one or more business processes or technical systems. 

Unlike the other data applications discussed above, it does not typically take a product engineering unit to ship a dashboard or report: a BI analyst can often do this in a business intelligence tool. However, the distinction between the two is blurring, as organizations look to deliver reports and dashboards into broader applications that are developed for customers or team members (look at how BI vendors all support embedded delivery of dashboards).

Generative AI-powered applications

The emergence of generative AI has dramatically increased the scope of what data applications can do.

While the focus of most data applications prior to the development of generative AI was on making decisions or supporting humans in making decisions,  the technology has the potential to grow the scope of decisions that a data application can support, or make itself. For example:

• Rather than simply optimizing which ad creative from a set of assets to use in an advertising campaign, a generative AI-powered data application might generate new creatives, that are better performing than any of the human designed examples, driving improved return on ad spend by growing the number of possibilities that the optimization is performed across 

• For the apparel buyers  mentioned above, generative AI could be used to suggest new products that would do well that the team might otherwise have missed.

How do companies organize to develop data applications successfully?

Companies that want to derive value from data are increasingly looking beyond business intelligence and forming teams capable of building, shipping and supporting data applications. 

These teams look like classic product engineering teams, but typically have additional skills in data engineering, data science, MLOps and LLMOps, alongside the more typical skills like front and backend engineering. 

This was a deliberate strategy employed by a fashion brand and Snowplow customer. Its Data & Analytics team spun up a number of product engineering units to develop and roll out key data applications across the business including:

1. An application that supports buyers with their quarterly planning (described above)

2. An application that helps the team figure out where and when to set up pop-up stores This is a very complicated thing to measure - it is not trivial for example to understand how many sales generated by the pop-up store cannibalize the permanent outlets / channels 

3. A content ranking engine to help the social team put out posts that are more likely to go viral 

4. A media optimization app that combines the media mix model with multi-touch attribution to help the marketing team allocate budget across different channels.

The key thing for the organization when implementing this strategy was that its data science team wasn’t just composed of data scientists, but it had access to all the required people (product managers, designers, front end and back end engineers) to design and build applications with UIs and algorithms that meet the needs of the different teams in the business. 

These teams were given 6-12 months to launch an app and measure its impact (i.e. the return on investment), and then if the application was shown to add value, they had the opportunity to iterate on these applications to drive higher returns. 

Data products enable data applications

Organizations that invest heavily in data applications can realize significant efficiency gains from investing in a well-governed underlying data set and associated technology so that product engineering teams have great source materials (data + supporting tooling) for developing data applications. 

A big challenge in developing a data application is to ensure that it’s built on a solid underlying data set that:

• Provides all the information required. For example, if a data application uses machine learning, the data set should include plenty of highly predictive features

• Is of high quality (i.e. the data is accurate and complete). Otherwise, dashboards will give a false picture of reality, decision support systems will lead people to make the wrong decisions. In addition, automated classifiers and recommendation engines will misclassify or recommend inappropriate items

• Provide the data continuously with the required latency, schema, encoding, etc.

A well-governed, well-documented set of data products, as per my prior post, provides an excellent foundation for building data applications. The difference between data products and data apps is not always well understood - let’s share a typical workflow that highlights the difference:

• To create a data application, product engineering units do not have to start at ground zero. They can build on one or more data products that meet their needs

• They may find an existing data product that meets their needs in the data catalog (which has to be kept up-to-date and accurate) 

• If not, they can create a new data product - either from scratch (e.g. by sourcing new data) or from other existing data products. By publishing this data as its own data product, future teams that need a similar data set can build from the same basis, rather than having to start from scratch themselves

• As part of the data product, it should be clear what SLAs the data adheres to, and the application developers have confidence those SLAs will be maintained going forwards

• The product engineering team can engage with a data product manager, who liaises with the teams that produce the data. The data product manager explains to the data producers who’s using the data for what, so they can take this into account both when delivering the data on an ongoing basis and managing the roadmap for it. 

So, how should organizations get started building data apps and data products?

Here are some guiding principles for getting started with building data apps and data products:

Step #1

Organizations should first build data applications before investing in data products. 

This might sound counterintuitive: after all, it is much easier to build a new data application if you already have one or more data products to build on. 

However, it is always best to start with the business value you want to create - i.e. the data application you believe is going to move the needle for your business, and then work backwards to the data product(s) that are necessary to support this data application. If they don’t exist yet, you now have a business case to support building them.

Step #2

Make sure that the potential value to be generated is big. Building any kind of application, let alone a data application, is a significant endeavor. So make sure you pick a use case where there’s real value to be realized.

Step #3

Assemble the right team to seize this opportunity. You need the right combination of product management, data engineering and front- and back-end engineering professionals. 

Step #4

Give the team the space to execute. Like the brand I mentioned earlier that gave their teams 6-12 months to make a difference.

Step #5

Publish any data products developed as part of the data application. It is likely that they  will be valuable to other teams. The team building your first data application should set a good example in terms of creating the underlying data products and socializing them across the business.

In the last five years, AI has created opportunities for companies to use data in radically new ways. At the same time, the proliferation of digital platforms means organizations can create and work with entirely new data sets. 

Treating data-as-a-product is a new approach to managing data that has emerged in recent years. It goes a long way to solving the challenges of driving self-service, especially in today’s world where data consumers have the ability to do so many interesting things with so many different data sets.

In this post, I’ll examine why it’s so important to treat data like a product. In the follow-on blog post, I’ll look at how treating data-as-a-product supports the realization of value in data applications.

Image courtesy of Dall-E

Subscribe now

Why treat data-as-a-product?

Data is not oil

Data is valuable. The world’s leading companies use data to differentiate themselves from the competition. 

The best companies in the world are “democratizing” data so that every employee has access to lots of data that gives them up-to-the-minute insight into what’s going on in their business (e.g. what are our sales, who are we selling to, how profitable are those sales, how were they generated, who bought, what type of customers they were…). This gave rise to the idea that data is an asset. 

However, organizations face a wide range of challenges in realizing the value of their data, even when adopting best-in-class technology stacks (data platforms), such as cloud data warehouses, modern BI tools, AI tools, data transformation tools etc.

Enabling end users to self-serve data is difficult

It’s likely you’re keen to enable end users in the different lines of your business to “self-serve” on data:

• For end users, self-service is empowering - they can develop their own insights, make better decisions and build their own data applications to drive more impact and be more successful

• For management, the ability to self-service means that the quality of decision making across your organization is improved 

• For data teams, they can drive much more value by enabling a greater number of people to derive value from the data than if they had to create that value themselves - the data team is not constrained by their own size

For this reason, self-serve has been a big topic in the data industry for decades: business intelligence solutions emerged in the 1970s and 80s to enable different people across the organization to analyze the data in the data warehouse themselves. 

More recently, semantic layers and data catalogs have emerged as tools to support self-service.

Despite all this, self-service is still difficult. To enable self-service, it turns out that it isn’t enough to give end users access to data and tooling to slice and dice the data.

Data consumers need more than just data and tooling to be able to self-serve effectively

They need to know what data is available

Data discovery is a significant challenge in most organizations: there is typically a lot of data. That’s why data consumers need some kind of catalog that clearly shows what data is available so they can find what they’re looking for themselves.

They need to know how to access the data

It may not be enough to know that there is data available in a particular table, API, event stream, semantic layer, feature store or analytics tool. 

Data consumers might need additional instructions on how to access the data, including the ability to gain access to logins / permissions.

They need to know what the data means

To actually use the data to draw insight, engineer features or deliver a dashboard, data consumers need to understand and interpret the data. 

This is only possible if they understand what the data actually means, which in practice often means how it is generated (i.e. what triggers a new line of data, and the rules for setting the different cell / property values). This is typically referred to as the semantics of the data.

They need to know the quality of the data 

The data consumer requires a certain level of data quality - how high this level is depends on how they plan to use the data. If the data is going to drive an AI model, especially for a critical decision (e.g. fraud detection), the quality bar is set much higher.

If the quality is too low, the data consumer should not waste her time trying to use the data to solve her problem.

They need to know any SLOs / SLAs associated with the data

If a data consumer wants to build a production application that uses this data on an ongoing basis, she needs to know whether she can rely on the data being delivered within a certain timeframe, for example. This is particularly important for real-time and near-real-time applications. 

To enable self-service, data needs ownership

All the above information is metadata, i.e. data about the data. 

This is information that hopefully lives in a data catalog, but someone needs to populate the data catalog. 

Someone needs to own the data and be responsible not just for the entry in the data catalog, but also for ensuring that the data continues to meet the standards set in the data catalog (and that changes to the data are reflected in the catalog).  

Of course, we want to automate the population of as much of the metadata in the catalog as is possible. But there is always some, critical metadata that a human being will need to manually populate – or example around the semantics of the data. And there needs to be at least a designated individual responsible for ensuring that the data is as described.

Data teams are not in a position to own all the data

If they are the only owners of data, they become a bottleneck

In many organizations, the responsibility for delivering data to the business (and managing assets supporting the use of the data, such as data catalogs, semantic layers and business intelligence tools) lies with a central data team that is charged with driving data usage across an organization. 

The amount of data that is democratized around the business depends on the capacity of the central team.

Data teams are often not in a position to own specific data sets - especially data sets that live upstream of the data warehouse

The data value chain is long and opaque, making it difficult to understand who owns what. Often the data that is “democratized” in a business is the “gold” data in a medallion architecture. 

Illustration of medallion architecture by Databricks

• Raw data from different sources is landed as bronze data

• This is cleaned up into silver tables, which are intermediary building blocks

• Silver tables are then computed to deliver “gold” tables - business-level aggregates that are optimized for direct consumption by individuals in the different lines of business

In this setup, it is common to have a central data team responsible for the flow of data from bronze to silver and to gold. By extension, this makes them responsible for the gold tables.

However, the management of the data upstream of the data warehouse might be completely opaque, with a variety of tooling used to bring in data from different source systems, and multiple people responsible for the data quality of each of those systems.

For example, anyone who’s had to work with Salesforce data in a data warehouse will know this can be incredibly challenging. 

This is because it’s hard to get sellers and other end-users of the tool to adhere to high data quality standards when entering data. 

Consequently, it becomes hard to successfully process and transform the Salesforce data into something useful from an analytics perspective in the data warehouse, because the structure of the data is not optimized for analytics or insight. 

Try getting someone on the data team to compel the sales team enter data into salesforce in a specific way to make working with the data easier: it does not work.

Data teams might not be the right people to own data sets downstream of the “gold” tables

It gets more complicated. End users who want to self-serve the data are likely to explore the gold tables. 

But then they might take one or two tables and combine them to produce a useful view on the data. Or they might not find what they want in the gold tables, and work back to the silver tables, because they want to drill into a level of granularity that isn’t available in the gold tables. They’ll then take that information and combine it with data in other tables. 

A third user might find that combined view helpful, and then take those tables and combine them with some other tables. Pretty soon you have a spaghetti of data pipelines and tables, where it isn’t clear:

• Who created which table

• Why they created each table

• What the level of governance is on each table

• What the upstream dependencies are on each table

• What each table means

• What each table is used

• What SLAs (if any) are offered around this table

Image taken from https://www.technologyreview.com/2021/12/06/1040716/evolution-of-intelligent-data-pipelines/ 

To be clear, it’s great that teams in the lines of business are building new data tables that are not only helping themselves but also supporting other folks in the business.It’s also great that they’re building them on top of well- governed gold tables. 

But asking the data team to keep track of those tables and “own” them is not going to work.

From data-as-an-asset to data-as-a-product

One solution to the above challenges is to treat data-as-a-product:

1. Data tables have specific owners. (You can call these people “data product owners / managers” - note that in many orgs today, there are “shadow” data product managers i.e. people with other job titles but who are held responsible for one or more data sets.)

2. Different people own different data sets. Ownership is de-centralized

3. Data sets (e.g. specific tables) have customers – people that use the data to do stuff, including doing analysis, powering data applications e.g. fraud detection engines, dashboards

4. Those customers have needs that evolve

5. The data product owners / managers are responsible for meeting the needs of the customers of the data tables. This means they have a way of keeping track of who is using what data. It also means anticipating those changing requirements

6. The data product owners / managers have teams (e.g. with data engineers, analytics engineers, data scientists) so that they can build what is required to meet the needs of their customers

7. The data product owners / managers are responsible for the upstream pipelines that deliver that data set - unless they’re consuming from a different data set with a different data owner. In this case, they are a customer of that data set, with a dependency on it 

8. The data tables have a roadmap, worked out by the data product owner, to meet the needs of the customers 

9. As part of her commitment to meeting the needs of customers, the data product owner is not just responsible for the delivery of one or more data tables, but all the additional things that a customer would need to be successful with the data. This might include:

   1. Metadata i.e. entries describing the data in the data catalog

   2. APIs and UIs for accessing and working with the data

Multiple people and teams can create and publish data sets

Treating data-as-a-product is a valuable way to help organizations democratize data by ensuring that there are people with the right mindset and skills to facilitate the publishing and usage of different data sets by different stakeholders across the business.

However, if the data team is the only team capable of publishing and managing data tables for the rest of the business, then it will become a bottleneck for data democratization.

Instead, if many more teams in an organization can be empowered to publish and manage data sets for the rest of the business, then that bottleneck will be removed. 

Instead of competing on “how many teams use data to drive decision making”, companies can start competing on “how many teams are creating data sets that are adding value to the rest of the business”.  

Teams need to be incentivized and empowered to publish and manage data sets that are valuable to the rest of the business

This is a difficult problem to solve. But it can be solved.

Many teams need to create data sets for their own internal use. For example, a product engineering unit might create a data set that describes how users engage with their product so that they can use this data to improve the product. 

What if they had an incentive to publish and maintain this data set as a product that can be used by other people in the organization? 

Maybe it’s important for the customer success team to understand how different people are using the product and tailor their success plan accordingly? For the marketing team, the data could be useful when it comes to segmenting customers for specific campaigns. 

The key is to provide different teams with technology that makes the process of publishing and then managing data sets as a product as easy as possible. This lowers the cost to individuals and teams to create and manage data sets, empowering more people and teams to do so, thereby supporting the whole organization to develop and use more data sets across the business.

Data contracts support treating data-as-a-product

Data contracts are powerful technologies that enable individual teams to create and manage data-as-a-product. 

Data contracts are  machine-and human-readable abstractions that describe what each data set is (I.e. contains all the relevant metadata that a data consumer would need to successfully self-serve on the data.) 

Further, data contracts are enforceable, i.e they don’t just describe how the data should be, but can also be used to provide some level of assurance / guarantee that the data meets the specification given. 

For any data set we want to create, we can define a data contract that formally specifies:

• Who owns the data

• What the data means. In practice - how the data was created / manufactured

• How the data is structured (i.e. the schema)

• Where the data can be accessed. (From which API, via what type of protocol and access method(s))

• SLAs for delivery and access

• SLAs for data quality 

• Policy – what you may and may not do with the data

Data contracts can support organizations to effectively treat data as a product and thereby democratize and manage data more effectively, by providing the basis of a huge amount of automation which should make it easier for different teams to publish and manage data as a product:

• Anyone who wants to create a new data set can start by defining a data contract

• Technology (e.g. Snowplow) can then be used to automate as much as possible the process of creating (manufacturing) this data set, based on the definition 

• The contract can be used on an ongoing basis to ensure compliance with SLAs

  • Technology (e.g. Snowplow) can alert on breaches of the SLAs, e.g. if the data does not match the schema or the latency of the data has breached the SLA)

• The contract can be versioned so that people understand the evolution of a data set over time

• The contract can enable interoperability - for example a data contract can be used to facilitate the integration of a Snowplow data set with: a data catalog, a BI tool, a semantic layer, a data science toolkit, a user permissions and access management system. The contract can be used, for example. by access control systems to ensure that only the right people have access to the data set, or by other tools to automatically expire or anonymize data after a certain period of time

I wrote more about data contracts in my Substack post, ‘What is, and what isn’t, a data contract’.

So, what is a data product?

Now that we understand what it means to treat data-as-a-product and why we should do so, let’s come back and think about what a “data product” actually is.

There is a real lack of consensus in the industry about what a data product is.  

Some commentators associate the definition of data products with treating data-as-a-product, whilst others thinking more expansively about data applications (i.e. all the different types of applications you could build that utilize this data - more on this in my next post.)

My strong preference is that we define a “data product” as a data set that is treated as-a-product, for three reasons:

1. Treating data-as-a-product is an incredibly important and powerful shift in the way we think about and manage data 

2. We already have a perfectly good way to refer to data applications i.e. “data applications”

3. It is helpful to distinguish between “data applications” and the “data sets that are treated as products” i.e. “data products”

So if a data product is a data set that is treated as a product, what are the things we would expect to make up that data product? 

• The data itself 

• All the metadata required to enable people to self-serve on the data (discussed above)

• Additional metadata, e.g. who owns the data set

• A data contract. This would be a machine-and-human readable version of all the metadata.

A data product then is a well documented data set with an explicit owner, who manages it as a product, and an associated data contract. 

Rather than thinking about data as an asset, organizations should instead be looking to build data products, as an effective approach to enabling much wider, self-serve usage of data across the organization.

In the next post, I’ll take a more detailed look at data applications and explain how data products provide an excellent foundation for developing more effective data applications.

But as data contracts become more widespread we’re seeing the concept being watered down as more and more people jump on the bandwagon. (This might have reached its nadir with Hightouch’s launch of “data contracts”.) So now feels like a good moment to review what are the critical components of a data contract and why. 

Illustration by Midjourney Pro

What are the critical components of a data contract?

Ownership

In order to use data effectively in an organization, there must be clear ownership for each stage of the data value chain. This is  difficult to achieve in practice, as many organizations overload their central data team with responsibility for the entire data value chain. Data contracts are designed to solve this bottleneck in data teams , by providing a clean way of assigning ownership for the different parts of the value chain to data producers and consumers.

Ownership is therefore one of the fundamental pieces of metadata that has to be captured in any data contract. A data contract should not just describe what standards the data will meet at different points in the data value chain, but who is responsible for ensuring that the data meets that standard.

It is not uncommon to not only identify the owner of the data set described by the data contract, but also the different ways to contact the owner and get support in the data contract itself.

Schema

 To consume data on an ongoing basis, you need to know how that data is structured. (If you’re performing a one-off analysis, you can spend the time figuring out the structure. But if you want to keep a data application consuming a data feed on an ongoing basis, you need to know that the data that will arrive will conform to the same structure as the data you’ve consumed to date.) Schema is therefore a key element to any data contract. 

This is the element of data contracts that everyone remembers, however lax their definition of a data contract is. At Snowplow, we’ve been supportng this element since 2014 when we first introduced self-describing JSONs and then Iglu, our schema registry. However, as we’ve seen above with Ownership and we’ll see below, a data contract needs to contain a lot more than just a schema. 

Semantics

To gain insights and draw conclusions from data, it is not sufficient to know the structure of the data (i.e. the schema), you also need to know what the data means (i.e. the semantics). In many organizations, there are typically a handful of people who are known as “experts” on particular data sets because they have lots of experience working with those data sets, they understand exactly how the data was created, and as a result are able to use those data sets to develop answers, insights and applications much faster than their colleagues. The difference between them and the rest of the company comes down to a deep understanding of the semantics of the data. Let’s bring this to life with a few examples:

• If you have a data point that describes how long someone ran for, you not only need to know that it is a numeric field type, but also that the units are in km rather than miles, for example. If the data changes, so that at a certain point it starts reporting in miles rather than km, or some of the data is in km and some is in miles, that is a problem that needs to be addressed as part of the data processing.

• If you have Salesforce opportunity data, it’s not enough to know that a particular field represents “stage 4 in the opportunity funnel” - you need to know the definition that sales people use to categorize that opportunity in stage 4, and how that definition might vary between different members of the team and the team over time. 

• If you have a data point that describes the number of sessions that a user has engaged with your applications, you need to know how those sessions are defined, and any variation in that definition across different applications and platforms and over time. 

A data contract can’t just be a commitment that the structure of the data is not going to change within certain boundaries - a similar set of guarantees is required for the semantics of the data. This is a lot harder to monitor and enforce than a schema - I’m planning to dive into this in a dedicated post shortly, as it is an important topic that deserves its own treatment. 

Delivery and access SLAs

If you are going to build a data application that consumes some data, you need to know where and how you’re going to access the data, and how readily the data will be made available. The data contract should include information like:

• Where the data can be accessed (i.e. physical location) e.g. a table in a warehouse or data lake, or a Kafka topic, for example.

• How the data can be accessed e.g. via an API.

• How quickly the data is made available (latency). This is going to be essential for powering real-time use cases (e.g. fraud detection, or dynamic pricing).

Data quality SLAs

If you are going to build an application on top of a data set, you need to have a certain level of assurance around the quality of the data.

In an ideal world, a data quality SLA would provide assurance around:

• The accuracy of the data i.e. to what extent when the data represents that something has happened, that it has really happened, within a certain degree of accuracy. For example, if the data says some individual completed a run, then you have a certain degree of confidence that they really completed a run, and that the length of the run, the time taken and the time completed are accurate to a certain, specified level.

• The completeness of the data i.e. if the data says that an individual did not complete a run (because there is no indication in the data that she did), then how confident can we be that she actually did not?  

In practice these are very hard if not impossible things to specify categorically: we generally don’t have a way to check our data against the real world it is supposed to describe - the data is our best representation of that world. So most data contract implementations instead specify tests that can be conducted on the data - either on individual lines of data, or on the aggregate data set, to build some lower level of confidence in its accuracy. For example:

• A test could be carried out on data describing a run completed by an individual to test that the speed is within certain bounds, and that there’s no data describing the individual performing a different activity (e.g. swimming or sleeping) at the same time

• Tests could be conducted to ensure that the distribution of the data is as expected

Whilst there are significant technical challenges to specifying better data quality SLAs, as with semantics I am optimistic that there is significant scope for technology innovation here. I look forward to sharing more in a future post. 

Policy

It is very common in an organization to have multiple different data sets with different rules related to what can be done with the data, based on a combination of legislation around:

• The use of personal data and the basis on which this has been collected

• The use of AI to make decisions, especially as this relates to the provision of services e.g. credit to individuals

It is therefore essential that data contracts include metadata that

• Describes exactly what the associated data can and cannot be used for.

• Identifies any fields that are personally identifiable.

• Identifies any fields that describe decisions made by an AI that are subject to regulation, and if so what standards those decisions need to meet. 

• Data retention policies (how long this data can be stored for)

• Data anonymization policies. (Do any fields need to be anonymized or pseudonomized?)

Enforcing schema is a necessary first step, but by itself not sufficient, to enable data contracts

To really drive data quality and improved democratization of data in an organization, all of the above elements need to be incorporated in data contracts.

Towards an open standard for data contracts

There are a number of very promising initiatives to develop open standards for data contracts - the ones we are tracking are:

• Datacontract.com by Jochen Christ and Dr. Simon Harrer 

• The Open data contract Standard (ODCS)

• PayPal data contracts by Jean-Georges Perrin

As an open standard for data contracts emerges, we are looking forward to supporting it at Snowplow.

The recent debates about data contracts are really debates about data governance, because data contracts are tools for defining and enforcing data governance. (So the debate decomposes into two issues: should we be governing data more tightly, and if so, what is the best engineering approach to doing so?) 

In this post I want to unpick:

1. Why is data governance having a moment? Why are so many people arguing for more data governance, now?

2. What is the right level of governance to apply to different data sets? We often assume that more governance is better. This is not correct - I think there are very practical / pragmatic steps teams can take to figure out what the right level of governance is, for a specific data set, at a specific point in time, and evolve their approach as the way that the data is used in the organization changes.

What is data governance?

There are lots of definitions of data governance that are accurate (on the “what” of data governance”) but not helpful if we want to understand why data governance is valuable:

• “A system of decision rights and accountabilities for information-related processes, executed according to agreed-upon models which describe who can take what actions with what information, and when, under what circumstances, using what methods.” (Data Governance Institute)

• “The exercise of authority and control (planning, monitoring, and enforcement) over the management of data assets.” (DAMA International)

• “The specification of decision rights and accountability framework to ensure the appropriate behavior in the valuation, creation, consumption, and control of data and analytics.” (Gartner Glossary)

I think it’s helpful to think about data governance in terms of control. The more strictly we control data through its lifecycle, the higher degree of assurance and confidence we have that it is used in a clearly specified set of ways. We can differentiate between how strictly we control:

1. How data is created, processed and delivered to the different consumers (applications and people) that want to use the data.

2. How those data consumers interpret and use the data. 

If we tightly govern (1), we end up delivering data to consumers with a high level of assurance around what it means and how accurate and complete it is. If we tightly govern (2), we have a high degree of assurance about how the data is interpreted and used. 

Why is data governance having a moment? Why are so many people arguing for more data governance, now?

A number of factors drive the need to govern data more tightly:

1. Regulation forces us to govern data more tightly, so we can represent to regulators that the way we use data is compliant. 

2. Data socialization and democratization drive us to govern data more tightly. That’s because the more different data consumers are able to self-serve on the data, the higher the risk that someone somewhere misinterprets the data, or uses the data in a way that we are not comfortable with. (Maybe from an ethical standpoint, or a methodological standpoint.) A nice example is from accounting: when public companies produce accounts, the processes for calculating the line items in the financial statements need to be very tightly governed, to ensure that any analyst and shareholder interpreting the data can effectively do so without being misled, and to make it easy to compare line items between different companies safe in the knowledge they are comparing apples with apples. Financial audits are checks on that process.

3. Criticality of use cases drives a need for more tightly governed data. If we are using the data to make a life or death decision about how to treat a patient, or whether and where to launch a missile, we want much higher levels of assurance about both the data and the way the data is used to reach the decision than if we’re targeting an advert or plotting traffic trends over time. 

Given the above, it’s not surprising that data governance is an increasingly hot topic:

1. Data and the use of data is being increasingly regulated.

2. Organizations are socializing more and more data. In the case of behavioral data, for example - a data set that was once produced by one team (the digital team) and consumed by the same team (the digital team) is now produced by multiple teams and consumed by multiple teams across the business, for a whole host of insights and data applications including personalization, customer segmentation, product analytics, lead scoring, churn reduction and more. 

3. Organizations are increasingly using data not just to build reports and power analysis, but to actually drive business critical data applications (e.g. pricing optimization or fraud prevention, to give just two examples).

Many organizations have made significant progress socializing more and more data sets, and using data to power more business critical applications. Previous efforts to do this have been gated by technical challenges around collecting the data in a single location, processing it at scale and velocity, and delivering it into tools that enable data consumers to explore and activate it. But as the toolset for working with data has come on leaps and bounds (with cloud data warehouses, new tools for building and running data pipelines, next generation AI and BI tooling etc.), the constraints have shifted: now it is the quality, consistency and level of assurance around the data itself that is holding organizations back. So an increasing amount of attention is shifting to governance to deal with this new constraint. So governance, observability, lineage and semantic layers have become hot topics.

Data governance costs

This is an obvious but important point: there is a real cost to implement data governance. It is cheap to slurp up data at scale and bung it in a data lake. It is harder to be precise about the data that we want to create: to tightly specify the engineering processes that will be used to generate, process and deliver that data, and document and rigorously enforce those processes. There’s a return on that investment though: it will be easier to socialize the data that is created, because there is a high level of assurance that the data means what it says, and it is clear how to correctly use the data to drive insight and value. This makes it easier for data consumers to build on, and easier for security teams to ensure compliance with regulation.

So for any data set there’s a trade off: what’s the right amount of resource to put into governance, to drive an appropriate level of return? Too much and we’re wasting time and effort governing data that might not being used to drive value. Too little and we expose ourselves to unnecessary regulatory risk, and make it too hard for data consumers to build value creating applications on top of the data.

Data governance is not one size fits all: different data sets require different levels of governance

The right level of governance is going to vary data-set by data-set, because we use different data sets in different ways. Data sets that are more highly regulated, more widely socialized and used to power more critical applications, need to be more tightly defined than data sets that do not. Within an organization there will be a spectrum of data sets, with very tightly governed data sets (e.g. master data, financial data) on one end of the spectrum, and completely ungoverned data sets on the other. (E.g. data related to experiments performed by individual teams, that is only consumed by those individual teams, for the duration of those experiments.) Most data sets will sit somewhere between these two extremes.

Data governance is not one size fits all: the level of governance for a given data set could well change over the lifecycle of the data

As the uses for a particular data set change, the level of governance that needs to be applied to that data set will need to evolve. 

Consider the example of data that describes how a customers engage with a purchase funnel. This data might have originally been created by a product team that wants to optimize the purchase the funnel. The team is working rapidly to experiment with different approaches to purchase, and as they do, they’re running multiple experiments and evolving the way that they generate this data very rapidly. Given they are the only people using the data, that is fine. The data can be completely ungoverned. (Or more precisely, informally governed - if someone in the team isn’t sure how to interpret the data, she can ask her team mates, who have all the relevant knowledge.)

Now consider a second team who are working on personalizing the user experience. They want to use the data to understand whether a personalized experience makes it more likely that a user will progress down the purchase funnel. Further, they want to use the data to drive the personalization - if they see that a user has progressed down the purchase funnel with particular items in their basket, that’s valuable product affinity information they can use to build features to power their personalization models.

At this point the level of governance needs to increase. There are now two teams that are consuming the data. Further, the second team is using the data not just to drive human decision making, but to drive a production data application. They could rely on informal collaboration to ensure that their use of the data is correct as the first team evolves the data. Or the organization could formalize the specification for the data, and the enforcement of the specification, such that the second team (and a third if it comes along) can self-serve on the data based on the specification. The latter is going to make more sense as more people consume the data, and as more business critical applications are built on top of it. So a data set that started off as requiring little governance, ends up needing more and more as it is adopted by more and more data consumers in the organization.

Seen in this light, the “pain” of discovering that a data set is not sufficiently governed is a good thing - it suggests that the business is using the data in more sophisticated ways, building new use cases and data applications that (hopefully) deliver real value for the business.

So how should we determine what is the “right” level of data governance for a specific data set?

This is more art than science, but I’d suggest that “pain” is a good guide. If your organization is experiencing pain because data is not sufficiently governed - if there are things you’re trying to do at your organization that are being held back because of a lack of assurance around the data, then that’s a strong signal that it’s time to up the level of data governance. Conversely, if you’re not experiencing that pain, now’s not the time to invest in governance.  

This recommendation is at odds with a lot of folk I see advocating for data teams to invest in data governance early. The thought here, I believe, is that by investing in data governance as early as possible, data teams can avoid the pain of discovering data quality is a blocker to effectively build a data application or use case down the line. This is misguided. It only makes sense to invest in data governance if it’s going to provide value - if it’s going to reduce regulatory risk, power data socialization or enable more powerful use cases. If not - it is wasted effort. It might be that the effort is not wasted if there is a need at a later stage to socialize the data or build more critical applications on top of it - but at this point we’re gambling: the number of data sets in a modern organization is enormous - it’s crazy to go governing all of them in the hope that there’s a payback down the line. 

I’m old enough to remember when the conventional wisdom was to collect all the data you can so that when you decide you want to do something with with it, you already had it. That didn’t work because when you came to use the data, you inevitably discovered that it wasn’t fit for purpose and then all the expense of collecting it was for nothing. It is the same thing with data governance - you can spend lots of time and effort tightly governing data, but unless you know how that data’s going to be used, you run the risk that that effort (which is more costly than just collecting the data) will be in vain, and the data still wont be fit for purpose, or worse, there is no value-creating data application or use case to be built.

So if you’re not experiencing the pain of ungoverned data - don’t bother investing in data governance. There’s some other constraint holding back your data ambition (or something constraining your data ambition) and that is where your focus and effort should be. And when you start to experience that pain - take that as a positive. It means your organization is evolving their use of data, and now the investment in governance will pay off.

Why all the fuss? In this post, I will argue that Data Contracts are:

1. Very obviously a good idea (and one that should not be controversial)

2. The reason this sensible idea is contentious today in the context of the Modern Data Stack, is that practitioners are stuck in the “data is oil” paradigm i.e. assume that the data is extracted, rather than deliberately creating data

Why Data Contracts are obviously a good idea

Let’s take it right back to the basics. In a modern organization there are many teams that consume data: to take just one (broad) example, data describing how your customer interacts with your digital products is going to be consumed, by:

• Your product team, who want to understand how those customers engage with different products and services, and how changes to those products and services drive changes to customer lifetime value

• Your customer team, so they can better understand and serve those customers

• Your marketing team, so they can more effectively target and market to customers based on their interests, product affinities and where they are in the purchase cycle

• Your commercial team, who want to understand revenue and profit per customer and customer segment

The data might not just be being used for analysis and understanding: it might be used directly to power production data applications that might, for example:

• Personalize the customer experience

• Triage support tickets based on customer value (so that high value customers get a better, faster service)

• Automatically generate audience segments (e.g. customers with a high propensity to churn) for targeting with particular offers, driving better, more relevant customer experience, and higher performance from those campaigns

• Identify customers who look like they might be committing fraud or victims of fraud, and take remedial action accordingly

So we have a situation where the data produced by one team (the product team), is consumed by multiple teams. To make this more complicated, we have multiple teams producing data, all of which is consumed by multiple teams:

• Each product team / unit / squad is likely to produce its own data describing how users engage with that particular product or service

• Customer teams will generate their own data in their own CRM / loyalty / support systems

• Marketing teams will generate their own data in their own marketing tool that describe which campaigns and content have been served to who, and any subsequent engagement data (e.g. open and click through rates)

• Commercial teams are likely to generate their own data e.g. profitability models

The situation is a mess and the mess drives fragility. Any change to the data produced by an individual team has the potential to break multiple downstream reports and production applications owned by other teams. Worse, it is often completely opaque to a data producer which teams and applications are consuming the data they produce, and what assumptions and logic they have assumed is baked into the data. This means that even if they wanted to check with data consumers prior to making a change to the data produced they would not even know who to ask, nor have anyway of knowing what sorts of changes to the data are safe to make.

Data Contracts have been proposed as a straightforward solution to this challenge. Data producers commit to producing data that adheres to a specific contract, that typically defines at minimum, what schema the data conforms to, but can (and should) include other valuable information for teams that want to consume the data like:

• The semantics of the data i.e. what the rules are for setting properties to certain values. (We record transaction values in US dollars, we record a transaction when the payment gateway returns a successful response…)

• SLAs on the data (around accuracy, completeness, latency)

• Policy (what the data can and cannot be used for - this is particularly important for personal and health data)

• Information on who owns the data and how to go about requesting changes to it. (Data Contracts can be versioned so allow not just for the data to evolve over time with the needs of all teams, but provide a useful tool for enabling the controlled evolution of data in the business.)

Data consumers can review the contract, and then architect downstream applications that only make safe assumptions about the data i.e. assumptions that are explicitly captured in the contract. Then, multiple teams can build downstream data applications safe in the knowledge that the Data producing team will not make any changes to the data without first publishing a new version of the Data Contract. (Which should always be ahead of actually making any changes to the data produced.) Each data set (or “data product”) only needs a single Data Contract, and this contract can be used by all consumers of that data set.

So what is so objectionable about a data contract?

The most articulate argument against data contracts was made by Benn Stansil:

The way data contracts try to achieve this though—through a negotiated agreement—seems wrong on all fronts: It’s impractical to achieve, impossible to maintain, and—most damning of all—an undesirable outcome to chase…

It’s impractical to achieve because

A lot of data “providers” (e.g., software engineers who are maintaining an application database, and sales ops managers who configure Salesforce) can’t guarantee them anyway. If you change some bit of logic in Salesforce—if, for example, a team stops recording pricing information on the product object and starts recording it on the product attribute object—it’s hard to know how that change will be reflected in the underlying data model. Salesforce’s UI exists for exactly that reason—so that we don’t have to think about our entire CRM as a bunch of tables and an entity-relationship diagram. If administrators sign a contract to maintain a particular data structure, how can we expect them to hit what they can’t see?

It’s an undesirable outcome to chase because

We don’t want data providers to be worried about these contracts in the first place. Engineers, marketers, sales operations managers—all of these people have more important jobs to do than providing consistent data to a data team. They need to build great products, find new customers, and help sales teams sell that product to those customers. The data structures they create are in service of these goals. If those structures need to change to make a product better or to smooth over a kink in the sales cycle, they shouldn’t have to consult an analytics engineer first. In other words, data teams can’t expect to stop changes to products or Salesforce; we can only hope to contain them.

The key thing to realize is Benn’s exemplar for a data source is an export of Salesforce data into the data warehouse. He is 100% right that it is practically impossible to impose a data contract on data coming out of Salesforce. But he’s wrong that this means that data contracts are a bad idea. It is a bad idea to build production data applications (or important reports) on top of data extracted from Salesforce. Instead, you should be deliberately Creating Data to power your data applications, data that adheres to clearly defined and versioned data contracts.

What do you mean you should not build data applications on top of data extracted from Salesforce?

Fivetran, the vendor most commonly associated with ingesting data into the modern data stack, differentiate the following “sources” of data:

The Fivetran world view (shared by many in the Modern Data Stack) is that for any organization there is a finite set of sources (“data deposits” like “oil deposits”) where data can be extracted (like oil can be mined) and then loaded into the data warehouse (where later it will be refined into something valuable you can put in your car or power your ML application). The different icons on the left represent five different types of data source. Each type requires a different engineering approach, from Fivetran, to extracting and loading into the data warehouse. So this diagram is very useful if you’re a Fivetran engineer.

From the perspective of a particular organization or a particular team in the organization looking to get hold of data to power a data application, however, distinguishing potential sources of data in this way is not especially helpful. Taking a step back, data that an organization has resides in the applications it runs: typically in a database that is sat behind the application. If it is an application that the business is running itself, then it is likely to be possible to access the database behind a specific application directly, and replicate the data from there. (Using Fivetran potentially, or a Change Data Capture process, more generally.) 

If instead the application is provided as SaaS, like Salesforce, then it will not be possible to access the database directly: instead the data in the database might be exposed via an API, that can be regularly polled for data. This fits the data as oil metaphor, with a regular process setup to ping the API, extract the data, and then copy it into a set of corresponding tables in the data warehouse. It is the same data, conceptually, whether it is accessed via a RESTful API or via SQL (or any other type of API). The difference is in the access pattern, that reflects who is responsible for running the application.

The trouble with both of the above technical approaches (extracting data directly from databases or via published APIs) is that the data made available is a side-effect: it has not been deliberately created for a specific data application or team, it is data exhaust. It would be very unsurprising if over time the way data was structured in the database would evolve quite dramatically to drive performance improvements (e.g. as the application scaled over a larger user-base) or as the application evolves functionally. Ben Stansill is right that it is undesirable to ask a the application developers to consider not just the performance of their own application when designing those database tables, but also the performance and integrity of any downstream data consumers that consume the replicated version of the database. The APIs that sit ontop of most SaaS solutions are typically either built to reflect the structure of the database in those underlying databases, or to power specific data integration use cases. (Rather than generalized downstream data intensive applications.) So we should not be surprised that the developers of those APIs do not design them around the needs of downstream data consumers. This is one reason why so much work is required to wrangle extracted data into something useful. 

There is one type of data though that can provide a solid foundation for Creating Data optimized for downstream data consumers, data that can be governed by a contract. That data type is the third one listed in the Fivetran diagram, it is the humble “event”.

With “events” (i.e. Data Creation), it is straightforward to adopt data contracts

When we think of applications emitting “events” there are typically two types of application we think about:

1. Web and mobile applications that typically emit events for analytics purposes. (By integrating SDKs from vendors like Google Analytics, Adobe, Snowplow, Segment or Mixpanel amongst others). These events are emitted as HTTP requests.

2. Server-side systems that follow event-stream architectures. These are typically microservices that perform very specific functions, and communicate what they are doing to other microservices in the stack by emitting events into streams or topics rather than HTTP endpoints (e.g. in Kafka) and learn the current state of different entities that they care about by consuming the events emitted by other microservices in their ecosystem

In both cases, the application is deliberately creating data for use by other data consuming teams that are fundamentally decoupled from the teams responsible for the applications themselves: in the case of web and mobile product teams for the analytics team(s), and in the case of microservice applications for the other microservice application teams. 

It is pretty common in both cases to adopt data contracts to support this effective exchange: Snowplow has supported data contracts since 2014 , with Segment adopting them with the introduction of Protocols in 2018 and Adobe with the release of the Adobe Experience Platform. In the case of microservices infrastructure the Confluent team published the first version of the Confluent Schema Registry back in 2014. There is scope to extend all these technologies to power more comprehensive data contracts: ones that go well beyond simply capturing the schema and some simple semantics. But the principle of a data contract has been there a long time because it is the only way to ensure data integrity, in the case of both digital analtyics applications and microservices architectures.

This model of publishing data as “events” for external consumers is really powerful:

• It means applications deliberately compose and emit data: the creation of data it not a side effect. As a result, it is really obvious to any developer working on an application that if she makes any changes to the code that emits that event that will have an impact on downstream consumers, because the only purpose of the code to emit the data is to serve downstream data consumers.

• The emission is light-weight. Whilst it is not desirable to ask an application developer to optimize the design of a database to serve a downstream data application, it is entirely reasonable to ask her to write some code to deliberately emit data that meets a specific specification (the data contract) - especially when supported by SDKs that abstract away technical complexity, so the developer can focus on emitting the right data at the right time. (As specified in the data contract.). 

• There is already a well developed ecosystem of tooling for ensuring that the data emitted from applications as “events” is compliant with data contracts. The data contract can be used to derive the SDKs that are used to create the data, so that there is a causal link between the contract specification and the data emitted. Further, checks can be performed on the data as part of CICD and in production, to verify that the contract is not violated. (For example organizations running Snowplow use Snowplow Micro, a processing pipeline that is small enough to live on a container that is spun up as part of CICD, so that automated tests can be written to ensure that the right data is emitted at the right time with the right schema and semantics.) Today, it is easy for an application to integrate an SDK to emit events, debug the data emitted from a development environment, write automated tests to catch if the data violates the contract prior to the branch being merged to master and monitor the data in real-time to ensure it doesn’t deviate from the contract and alert on violations. 

When we are emitting events (i.e. Creating Data) it is easy to adopt data contracts to ensure data integrity. But when we are relying on extract and load processes to poll 3rd party APIs it is hard. (You can’t control Salesforce, you can only hope to contain it.) So given data contracts are so important, can we adopt the “event” model as the primary approach data used in production data applications?

We can adopt “the event” (i.e. Data Creation) as the primary approach for delivering data to production data applications

For any application that we have developed ourselves, we have the possibility to integrate SDKs and deliberately create data that adheres to data contracts. This is always preferable extracting the data from the underlying database (e.g. using change data capture) or from data archived from the data (sat on a file system), because the data is not a side effect, it is deliberately created and the process of creation can be governed by a data contract.

But what if we want to power our data application with data that resides in a SaaS application? Here I would make two observations and one prediction:

1. When you start developing a data application by considering what data you want (ideally), rather than what data you have (in different production systems), it is surprising how much of the data needs to deliberately created in your own applications. My favorite example is if you are building any sort of application that predicts the propensity for an individual to do something (like churn, upgrade, or buy a particular product) the time that an individual spends performing different related actions is nearly always the most predictive data: this is data that needs to be created in the applications in which the users perform those actions, and is never found in a CRM, that typically only stores much higher level (less predictive) facts about those same customers.

2. More and more SaaS providers are deliberately emitting data in real-time in an “event-like” way: it is very common for them to support exposing data as a stream of real-time webhooks, and even give users the ability to specify the structure of the payloads emitted and the rules for when those payloads are emitted. (Zendesk is a good example.) Here, Salesforce is the notable exception.

3. As more organizations start developing and running their own data-intensive applications in production, there is going to be real competitive pressure on SaaS vendors to provide their users with better tooling to create data from within them, the way that they can from their own applications. This has already started with webhooks, but needs to go further. (In terms of SaaS vendors publishing data contracts their webhooks conform to and providing functionality for users to customize those data contracts.) As every adherent to the Modern Data Stack knows, Martin Casado believes applications need to be reimagined so that they can be powered by intelligence built of an organization’s own data warehouse. I would argue that it is just as important that those reimagined applications enable the effective creation of data from the application directly into the data warehouse (including being governed by data contracts), as it is that they enable the flow of intelligence, built on that data, out of the data warehouse into the application.

Can we stop thinking of data sources based on the technical approach to moving the data (APIs, databases, events and files) and start distinguishing what matters (Data Creation vs Data Exhaust)?

I hope so but I fear not. But if you’ve made it this far I hope it’s clear why I believe this is the real distinction that matters to people who are building data applications. When data is created we can use data contracts to govern the process of creating and testing the data: and we then have a solid foundation for building downstream data processing applications. When we rely on data exhaust, we cannot.

Postscript: but we can enforce data contracts in CDC pipelines! (And Adrian and Chad have documented how do to this)

The day before I was due to hit publish on this post, Adrian Kreuziger authored a post on Chad Sanderson’s substack explaining how, from an engineering perspective, we can implement data contracts when relying on CDC for data ingest. This would counter my argument above, that data contracts only make sense when you’re relying on “event” data i.e. deliberately creating data, and undermine my argument that we should deliberately create data rather than rely on data exhaust.

In the setup Adrian describes, CDC is not being done directly on the tables in the database: instead an abstraction layer is introduced between the database and the place where the CDC events are generated. This provides a place where tests can be written that validate that the CDC events generated conform to the schema expected. Adrian describes how these tests can be run as part of CICD, failing a build where where a developer working on the application has accidentally made a change to the database that has caused a contract violation.

This achieves something really useful: it provides a way to catch releases that will break downstream data applications, and it provides those alerts to the data producing team, who are the right people to fix it. (If instead these tests were run in the data warehouse by an analytics engineer, the only possibility would be to add additional processing steps to take the data as ingested and try and wrangle it into the format as expected - which might work for schema changes but likely wont for semantic changes.)

But it is not nearly as good as the case where the data is deliberately created as an event emitted directly from the application code, because

1. It is likely to be a complete surprise to the developer. Until the build fails chances are that data contracts were not top of mind. (Or even under consideration) Were the data emitted as an event this would be explicitly visible in the code (it’s not a side effect) - so it is much less likely that the test would fail in the first place. 

2. There is additional work required for the application developer because the process of ingesting the data is a side effect that would not be required if the process for creating the data was explicit. Specifically, the application developer is not free to make changes to the application that change the structure of the underlying database as this will break the data contract. This will inevitably have some impact on development velocity. 

So maybe rather than a binary distinction between Data Creation and data exhaust, we should be thinking in terms of a more nuanced hierarchy, with:

• Data Creation (data as “events”) at the top, and data contracts driving both the creation of the data and the validation of the data

• Data extraction with data contracts enforced at source, next, so issues can at least be spotted and addressed at source, even if this generates more additional work for the application developers than data creation

• Data extraction with data contracts enforced in the warehouse, next, as better than complete chaos, but with the very significant challenge that the people who spot the issue (typically analytics engineers) have no context to understand the changes made, and are stuck with a limited toolset (SQL, dbt) to try and fix them after the fact

• Data extraction with no data contracts as a completely indefensible approach to application building

The metaphor of oil for data is incredibly pervasive. The metaphor infuses a lot of our core data vocabulary - you see it in common terms like “data extraction”, “data mining” and “data pipelines”.

The impact of the metaphor becomes particularly clear when we look at how data scientists work. In most organizations today, when a data scientist is charged with developing a new model to predict something, the first thing she will do is list a number of features she thinks are likely to be predictive. The second thing she will do is to look at whether there are data sources in the organization that she can extract and refine suitable data from, to engineer those features. In many cases she will conclude that many of the desired features she would like cannot be engineered because the required source data is not available. Her world is bounded by the data available at their organization - by the “wells” of data available to be extracted.

The impact of the metaphor is also clear if we look at any coverage of the Modern Data Stack. On the left of any diagram of the Modern Data Stack are the “data sources” - listed in much the same way oil firms create geological surveys of the world to identify the different areas with oil, suitable for extraction. “This is the (bounded) universe of places you can find data to replicate into your cloud data warehouse!”

Diagram taken from a16z blog post Emerging Architectures for Modern Data Infrastructure, https://future.com/emerging-architectures-modern-data-infrastructure/

In both cases, the assumption is that the data scientists developing the algorithms and models, or the users of the modern data stack being assembled, have

1. a limited number of data sources that they can tap, and

2. investment will need to be made in extracting, replicating and transforming the data to get it into a format from which it is possible to drive value from.

3. By implication, there is no data outside of those identified sources.

These assumptions are fundamentally wrong. Organizations can deliberately create the data that data scientists and other data consumers need, data that today those organizations simply do not have. More than that, organizations should deliberately create the data that they need: as Peter Norvig observed many years ago - “more data beats clever algorithms, and better data beats more data”: so it makes sense to spend at least as much time creating better data as developing better algorithms. In this post, I’m going to try and persuade you that organizations should invest in creating better data - and that investment can directly drive value and competitive advantage.

The limits of data extraction

The tooling for working with extracted data, and turning it into something that can be used to drive value, has never been better. The ecosystem that has grown up around the Modern Data Stack means that it has never been easier to build data pipelines that extract data from multiple sources (using tools like Fivetran, Airbyte and Meltano), transform it (using tools like dbt and Matillion) into a structure optimized for analytics and machine learning and deliver it into different analytics and activation tooling. Given there is so much value to unlock by adopting these tools and approaches, maybe it’s awkward to highlight that there are fundamental limitations to feeding this ecosystem of powerful tooling for driving value from data with data that is fundamentally limited. Unfortunately, we need to, because there are significant, fundamental challenges to working with extracted data, even with all that tooling available.

Data Scope and Granularity

Sometimes you want to engineer a feature or report on some behavior for which no data is available. Source data systems are finite: they have a certain amount of data with a certain associated scope. When a data scientist abandons developing a particular feature because the data is lacking, it is often down to the scope of the source data available.

Data quality

Data quality has become a bigger and bigger topic in data in general, as organizations start using data in more and more sophisticated ways, because more sophisticated data processing techniques require higher quality data. There has been an explosion of great tools to help companies build assurance in the quality of their data, solutions that include Monte Carlo, Datafold and Great Expectations. These tools provide structured approaches to check the properties of source data to ensure they are as expected, and build assurance that any subsequent processing steps don’t introduce new data quality issues. (This is really where these tools shine.) But by definition, there’s nothing these solutions can do to fix the quality of the source data - that is a given - and at best they can only flag likely issues. Data isn’t like oil - you don’t improve the quality of data by removing impurities from it - good quality data is created by ensuring that your process for creating the data is robust. For example, if you have a data set that describes the temperature of part of an engine over time, that data set is only as good as the equipment that was used to record the temperature and how well that equipment was used to generate the data set. If after the fact you find that some of the readings suggest the data isn’t accurate (e.g. they are suspiciously high), no amount of “cleaning” the data is going to resolve the issue that the data probably can’t be relied on. The quality of any derived data set will always be limited by the quality of the data at source. As the old saying goes, “garbage in, garbage out”… (Or as the English say, “you can’t polish a turd”, although it has not stopped lots of people in data spending lots of time scrubbing.)

Data Interpretability and Semantics

As organizations have started to use data to power more and more different applications and products, there has been an increasing understanding that data needs to be well understood and documented: it needs to be straightforward for a data consumer to understand what each row and column of data means. This has also led to another explosion of great technologies including semantic layers like AtScale, and Data Catalogs like Atlan, DataHub and Amundsen.

These solutions generally look to solve the problem of data meaning in the presentation layer: at the point that the data is made available to the rest of the organization. Meaning is effectively applied to the data (generally in the form of metadata) at the end of the processing. This means there are limits to how well understood the data delivered can be, because there is no solution to manage the meaning of the “source data” on which that presentation layer has been built. Instead, the metadata that captures the meaning has to be assembled by knowledgeable practitioners, generally after a long and tedious manual process unpicking the process for how the data was generated. Some of them work back from the presentation to provide visibility of the pipelines and processes that feed that presentation layer. Some of them use ML and social annotation features to support the process of curating the metadata and keeping it up to date. But none of them are able to work all the way back to the source systems where the data was originally extracted, where the data, and therefore the meaning originates.

When people in an organization look at data sources for potential data to extract, they typically find two types source:

1. SaaS applications like Salesforce. The data in these systems is typically extracted via APIs.

2. Databases within the organization. These have typically been built to support the applications that have been built by the organization themselves to support them doing business.

In both cases, each data set has been created to support an application - in the first case an application developed by a 3rd party vendor, and in the second, an internally developed application. 

Now, to understand what that data actually means, we need to understand:

1. How the application works. (In the case of Salesforce - an understanding of the concepts of things like accounts, opportunities and contacts)

2. How the application is, and has been used by the organization in question

This is not straightforward. Anyone who has ever tried to work with the old format Adobe “hits tsv” knows how much Adobe and implementation specific information is required to wrangle long lists of sProps and eVars. (Thankfully the format of data created by Adobe AEP today is infinitely superior.) Anyone who has tried to work with Salesforce data knows how sensitive the extracted data is to the particular ways in which individual companies have applied concepts like leads and contacts. In the vast majority of companies this sort of knowledge is tribal knowledge - stuck in the head of a handful of internal experts. Any data product managers, engineers, scientists that want to do anything valuable with the data will have to mine those internal experts to do anything valuable with the data from those sources.

As a result, it is very hard to draw meaning from the data from those source systems. This is one reason why driving value from that data is so hard: one of the reasons why so much time and investment has to be made to deliver a well curated data catalog or semantic layer.

Data structure

The structure of data extracted from different sources system is never optimized for machine learning or advanced analytics. This should not be a surprise - the data is structured in a way that is optimized for the application that it has been created to support, not the predictive model the data scientist is looking to build.

This is the second reason why organizations have to invest so much time today in data preparation: they are building data platforms on top of source data systems that do not reflect the plethora of data applications the data platforms need to support: instead they reflect the set of SaaS tools and internal applications that have been built to meet a completely different set of needs the organization has had to date.

In order to make that data usable for machine learning or advanced analytics, an enormous amount of data processing is generally required.

Managing change

One of the biggest challenges with relying on data extraction is managing change. A source system can change at any point in time: because the data extracted is a bi-product, it is generally completely opaque to the people that manage the application that changes that they make could fundamentally break any downstream data pipelines and data applications that rely on the data: none of those dependencies are explicit. If you are lucky, the actual data pipeline will fail, and someone can spot and address the issue. (A lot of investment is often made in creating CICD tests to try and do just this.) But if not, then the meaning of the data could change, without any obvious way for the people curating the semantic layer, data catalog or downstream applications to identify and adapt to the change. Building applications on this data pipelines that are fed by these sources is literally like building houses on stilts: the foundation is totally unsuitable for the structure it is supposed to support.

https://www.flickr.com/photos/79505738@N03/44203546932

Data Creation is a real and attractive alternative for data extraction

Organizations can and should deliberately create data to power machine learning and advanced analytics use cases, rather than relying on data extraction. This means that when a data scientist decides that a particular feature is likely to be valuable for a model they are developing, they should look at how the organization can go and create that data, rather than limiting their search for features to those that can be readily delivered using existing data sources.

The good news is that increasingly, organizations are creating their own data. To give just two examples:

Retail bank creates data to reduce fraud

A set of data scientists at a bank hypothesized that how an individual fills in a credit card application might be predictive of how likely that individual was to be applying fraudulently: for example, a user who tries pasting multiple specific social security numbers into the application form sequentially is likely to be making a fraudulent application. Similarly, a user who completes an application very quickly might not be a human but a bot - suggesting suspicious activity. 

This data can readily be created using open source technology like Snowplow (disclaimer: I am one of the founders of Snowplow) or alternatives. Indeed we have worked with a number of institutions to create data to enable features that are predictive of fraud, enabling those institutions to block fraudulent transactions and approve legitimate transactions faster. Behavioral data, describing what the individual submitting a transaction or credit application, has been found to be one of the most predictive indicators of fraud: incorporating this created data drives a step change in fraud detection performance. Yet this is a signal that many financial institutions ignore, because the data is not readily available to the data scientists: it needs to be created.

Insurance company creates data to measure exposure to flooding risk

Last year Europe saw widespread flooding. One of them realized that they did not have a way to accurately measure their exposure to flooding risk across their property portfolio. Although they had a list of addresses for all the properties that they insured, GPS coordinates for each property and accurate topographic maps covering Europe, it was not possible to tell how much damage would be done at different levels of flooding, because the perimeters of each property were unknown: instead only a single GPS coordinate given for each address, and sometimes this coordinate was not even within the perimeter of the property.

As a result, a data product manager assembled a team to solve this problem: their job was to create a data set that gave the perimeter of each property, so that the risk team could use that data to provide much better models for the impact of different levels of flooding on property damage across their portfolio. (By combining the perimeter data with the topographic data to figure out how much of each property would be submerged at different levels of rainfall.)

The data product manager and his team visited several hundred sites to directly measure the property perimeters, and then used that data to train and test a machine learning model that predicted those perimeters from satellite imagery. The resulting data made it possible for the risk team to drive a step change in modeling their exposure to flooding risk, and the insurance company is now exploring investing in developing many more data sets to drive improved risk modeling. 

One of the most exciting applications for machine learning, in data, is actually as part of the Data Creation process. Whilst in data we typically think of machine learning as a powerful tool for driving insight and prediction from data, it can be a useful tool in building a predictive data set to start with. At Snowplow, we see more and more of our customers and open source users recording the predictions that they make as behavioral data (in this case the behavior of the machine learning application) alongside behavioral data that describes the individual people who’s behavior powered those predictions. 

So if Data Creation is so much better than Data Extraction, why do so many organizations still rely on extraction for the vast majority of their data?

This is particularly surprising because as a whole, mankind is creating more data than ever.

Graph taken for Statista, https://www.statista.com/statistics/871513/worldwide-data-created/

It is just that by and large, data is being created by vendors of particular applications and products, rather than organizations looking to develop data platforms for their own internal uses. I think there are a couple of reasons for this:

1. The tooling for Creating Data is not good enough today. (I say this as the founder of a company trying to build better tools to create data.) For most data scientists that want to build a new data application and need to engineer a feature, it is easier to start with data that is already in the organization, however imperfect, than create that data from scratch. This is something that we are actively looking to change at Snowplow.

2. The “data is oil” metaphor is so pervasive in data that it simply does not occur to the vast majority of data practitioners. We have all been indoctrinated in the idea that data engineering starts with the data that an organization already has. To be fair, a huge amount of value has been unlocked by taking that data, bringing it together in a cloud data warehouse / lakehouse, and then building intelligence and activation on top of it. But so much more can be accomplished if organizations look beyond the data that they happen to have today, and start to deliberately create better data to power their data apps for tomorrow.  

When I left I became a Consultant - because that felt like a good place to learn how to use data to help businesses reason about things and use data to create value. I was very lucky early on in my career, working on a project with a manufacturer of complicated industrial goods, to use data that had been created for one purpose: recording the delivery schedules for different manufacturing plants, for an entirely different purpose: measuring the accuracy of the planning process. That analysis showed that actual planning cycles were significantly shorter than management had previously believed. It led to a wholesale restructuring of the organization. And so very early on I got to see the transformational impact of data, on a commercial context, with my own eyes. It was a totally different experience from anything I’d seen in the academic realm beforehand.

In the years that followed I got to learn how to use data to support companies developing strategies as well as improve operations.

In 2011 and 2012 I was very excited about what we now call behavioral data: the iPhone had triggered a wave of innovation in apps that permeated every part of our lives and that meant there were opportunities to create data that described how people behave: how they fall in love on dating sites, collaborate with coworkers to solve problems in productivity apps, learn on education platforms and spend their free time consuming media and playing games: data that was both broad (could describe millions of people) and deep (it described for each person exactly what she or he were doing, second-by-second and minute-by-minute). That, and the possibilities that big data frameworks like Hadoop and AWS services like EMR led me to cofound, with Alex Dean, Snowplow in 2012: an open source platform to create behavioral data at scale - one that any organization could use to create their own data. (At the time, organizations relied on tools like Google Analytics and Adobe, which prescribed a very narrow set of use cases on the data, and created all sorts of barriers to organizations to innovate on their own data.) Our goal was to set organizations - and data folk within them - free to innovate.

Ten years on the data space is unrecognizable. But it has only grown more exciting, there is so much more possibility and so much more innovation than I could have imagined back in 2012. Even better, being at Snowplow gave me the opportunity to work with a whole host of innovators in the data space and see first hand how advances in technology have been seized on to make fundamental changes to the way we are as human beings and the way organizations conduct themselves. It is incredibly good fun to think ahead, to the next 10 years, and speculate how they will evolve.

My intention is to use this substack to share and debate some of those thoughts: and widen the conversations I get to have around data, from the people I get to work with directly, to a much broader set of people like you, out there in the world.

Subscribe now