Expertise
Partnerships
Case studies
About
Insights
Careers
placeholder
    placeholder
    placeholder
    hero-header-image-mobile

    Semantic layers on Databricks for trusted enterprise analytics

    OCT. 5, 2026
    6 Min Read
    by
    Drew Minkin, Ph.D.
    A semantic layer on Databricks gives every dashboard, notebook and AI assistant the same definition of revenue, churn and margin, and that consistency is what makes enterprise analytics trustworthy.
    Most metric disputes trace back to business logic copied into dozens of BI workbooks, each one drifting a little further from the source. Data leaders already feel the gap, with 63% of organizations saying they lack or are unsure they have the right data management practices for AI. When the definitions can't be trusted, every report built on them inherits the doubt.
    The fix sits closer to the platform than most teams expect. Databricks SQL, Unity Catalog, well-structured repos and query history give you what you need to define a metric once, serve it everywhere and prove it's being used correctly. That toolkit got noticeably stronger in 2026. Unity Catalog Business Semantics reached general availability in April, Databricks open-sourced the core metric view implementation to Apache Spark, and releases since then have added materialization, window measures, parameters and relationships between metric views. The hard part is still the discipline around those tools.

    Key Takeaways
    • 1. Define each business metric once in Unity Catalog so dashboards, notebooks and AI assistants all return the same answer.
    • 2. Give every metric a single authoritative home, an owning domain team and a reviewed release path in version control.
    • 3. Use query history as weekly evidence that people consume governed metrics, and act on the gaps it shows.
    • 4. Pilot new Beta and Preview semantic features, such as materialization and metric view relationships, in development before relying on them in production.

    What a semantic layer on Databricks does for enterprise analytics

    A semantic layer is the governed set of business definitions that sits between raw tables and the tools people query. On Databricks, it translates columns into named metrics, dimensions and joins that every consumer inherits. When finance asks for net revenue, the answer comes from one definition, wherever the question is asked.
    Picture a retailer with three regional BI teams. Each calculates net revenue with slightly different return windows, so the quarterly review opens with a 20-minute argument about whose number is right. Moving that logic into a single governed definition ends the argument before the meeting starts, because every dashboard reads the same source.
    The value compounds once AI assistants start answering business questions. A language model querying raw tables has to guess what "active customer" means, and it'll guess differently each time. Gartner predicts that by 2027, organizations that prioritize semantics in AI-ready data can raise agentic AI accuracy by up to 80% and cut costs by up to 60%. Databricks makes a related argument, describing an ontology that ties governed business definitions to how the organization actually works as the context layer AI agents are missing. That's the practical case for doing this work now.
    Databricks is building its assistants directly on that layer. Genie Ontology, introduced at Data + AI Summit 2026, learns enterprise context from tables, queries, dashboards and the Unity Catalog semantic layer, then ranks sources by how authoritative they are. Genie One and Genie Agents (renamed from Genie spaces in July 2026) answer business questions on top of it, and governed Pages for business concepts, in Beta, give terms, acronyms and KPIs a single definition that feeds the same ontology. Every metric you govern well makes each of those assistants more accurate.

    How Databricks SQL warehouses serve governed metric definitions

    Databricks SQL warehouses are the compute that answers semantic queries, so their design decides if a governed metric returns in two seconds or two minutes. Serverless warehouses suit bursty BI traffic because they start quickly and scale out under concurrency. Separate warehouses by workload so ad hoc exploration never slows executive dashboards.
    A practical pattern is one serverless warehouse for scheduled dashboards, a second for analyst exploration and a third for Genie Agents, Genie One and other assistant traffic. Each gets its own scaling limits and auto-stop window, and each carries tags so cost lands with the team that generated it. Query tags, generally available since September 2026, take attribution a level deeper by labeling individual workloads by team, domain or dashboard, and they show up directly in system.query.history. When the CFO's board deck refreshes on Monday morning, it isn't queuing behind a data scientist's exploratory join across five years of transactions.
    One Beta worth watching is Lakehouse//RT, a real-time serverless SQL warehouse type released in June 2026 for sub-second reads at high concurrency. It isn't a default choice yet, but for governed metrics embedded in customer-facing apps or very high-traffic dashboards, it's worth piloting in a development workspace.

    "Moving that logic into a single governed definition ends the argument before the meeting starts, because every dashboard reads the same source."

    How metric views and Unity Catalog keep definitions consistent

    Metric views in Unity Catalog let you define a measure once, with its source table, dimensions, filters and aggregation logic, in a YAML specification you can create through SQL DDL or a point-and-click builder in Catalog Explorer. Any SQL user, dashboard, Genie Agent or alert then queries the measure by name, and external BI tools such as Power BI, Tableau and Sigma can read the same definitions. Because the definition lives in the catalog, it carries the same permissions, lineage and audit trail as the tables beneath it.
    Consider gross margin for a manufacturer with plants in four countries. The metric view handles currency conversion, excludes intercompany transfers and exposes plant, product line and month as dimensions. An analyst writes a short query using the MEASURE function, and the warehouse resolves every join and filter the same way it does for the finance team.
    Catalog placement matters as much as syntax. Teams that register metric views in a dedicated gold schema, with ownership assigned to a named data product team, get clearer accountability when a definition needs an update. Lineage then shows exactly which dashboards break if someone edits the logic, so changes get reviewed before they ship. External lineage, now generally available, extends that view to downstream Power BI and Tableau reports, and certified and deprecated flags tell analysts and agents which definition to trust.
    There's a constraint worth planning for. Window measures, generally available since October 2026, now handle rolling averages, running totals and semiadditive measures such as account balances, which used to require upstream code. Complex allocation rules and historical dimension tracking still belong in upstream transformation code. Keep the semantic layer thin and declarative, and push heavy reshaping into the pipelines that feed it.

    What's new in Unity Catalog business semantics in 2026

    What's new in Unity Catalog business semantics in 2026
    Databricks has shipped semantic features at a fast pace since business semantics went GA. Several are still in Beta or Preview, so pilot them in a development catalog and keep production definitions on GA capabilities until they graduate.
    • Materialization (Preview). Pre-aggregates metric views, refreshes them incrementally and rewrites queries to use the best available materialization, which keeps executive dashboards and agent queries fast without hand-built summary tables.
    • Parameters (Public Preview). Bind values at query time, so one definition can serve many variants, such as a currency or a fiscal calendar.
    • Relationships between metric views (Beta). Record how metric views join, including cardinality, so AI assistants and BI tools pick the right join path. This needs Databricks Runtime 19 or later, or a SQL warehouse.
    • Agent metadata. Synonyms, display names and formats help Genie and other assistants map a business question like "bookings last quarter" to the right measure.
    • Sharing metric views (GA). Since September 2026, partners and subsidiaries can query shared metric views with MEASURE through OpenSharing, so they use the same definition you do.
    • Domains (GA). Domains and subdomains organize tables, dashboards, notebooks and Genie Agents by business area on the Discover page.
    • Import from Power BI and Tableau (Beta). Brings existing BI semantic models into Databricks, which shortens the first migration step for teams with business logic trapped in workbooks.
    Openness is part of the story too. The core metric view implementation is being contributed to Apache Spark, and Databricks is among the 50-plus backers of Apache Ossie (Incubating), formerly the Open Semantic Interchange, a vendor-neutral specification for moving metric definitions between tools. Definitions you write today are less tied to one engine than they were a year ago.

    Databricks SQL compared with dbt for defining business semantics

    The main difference between Databricks SQL and dbt for semantics is where the definition lives. Metric views keep business logic inside Unity Catalog, close to governance and compute. The dbt Semantic Layer keeps it in version-controlled project code that moves across platforms. Many enterprises run both, with dbt shaping models and Databricks serving metrics.
    A bank running dbt for its transformation work might keep staging and mart models in dbt while publishing finance metrics as metric views, so Genie Agents and AI/BI dashboards read them natively. The split works when one team owns the handoff and documents which tool holds the authoritative definition for each metric.

    ConsiderationDatabricks SQL metric viewsdbt Semantic Layer
    Where definitions liveRegistered in Unity Catalog beside the governed tables they describeStored as YAML inside a version-controlled dbt project
    Governance and permissionsInherit catalog grants, lineage and audit logs automaticallyRely on project access controls plus the warehouse's own permissions
    PortabilityServed on Databricks, with the core implementation open-sourced to Apache Spark and Unity Catalog OSS support plannedMoves across warehouses, and MetricFlow has been Apache 2.0 licensed since October 2025
    AI assistant accessGenie Agents, Genie One and AI/BI dashboards query them nativelyNeeds an integration for each assistant or BI tool that consumes it
    Best fitTeams that want metrics served where data and compute already sitTeams that already manage transformation logic in dbt and want one codebase

    If you deploy metric views from dbt, check the adapter version first. Native metric view support in dbt-databricks is in progress but not yet released, so most teams today use a SQL model with CREATE VIEW ... WITH METRICS LANGUAGE YAML or a custom materialization.
    Pick one authoritative home per metric and record it in the catalog description. Duplicated definitions across both tools recreate the drift you set out to remove, just with better tooling around it.

    How to organize Databricks repos for analytics engineering teams

    Organize Databricks repos, now called Git folders, around data products such as finance, supply chain or customer. Each repository holds the transformation code, metric definitions, tests and deployment configuration for one domain. That structure gives every metric a clear owner, a review path and a release history you can audit.
    A typical layout separates folders for pipelines, metric view YAML, data quality tests and Declarative Automation Bundles configuration (the tool was called Databricks Asset Bundles until March 2026). Pull requests that touch a metric definition require review from the owning domain, and a CI job validates the YAML against a development catalog before merge. Deployment promotes the same bundle from development to staging to production, so nobody edits a metric directly in the workspace.
    Git folders also got more capable this year. Standard Git CLI commands, including pre-commit hooks, interactive rebase and LFS, entered Public Preview in July 2026, so the same YAML linting your CI runs can also run before an engineer pushes.
    Many semantic layer projects stall here, because the code structure gets designed after the first dozen metrics already exist. Lumenalta®'s MVP-led teams of people and agents set up repo conventions, review rules and promotion paths in the first sprint, with governance built in from the start, and the architecture stays client-controlled and portable so your team can run it after we step back. Retrofitting that discipline later costs far more than starting with it.

    "Ten governed metrics that every executive dashboard reads will build more confidence than 200 that nobody has validated."

    Using query history to govern how metrics get consumed

    Using query history to govern how metrics get consumed
    Query history shows who ran what, against which objects, on which compute and how long it took. The system.query.history table, generally available since September 2026, makes that record queryable across SQL warehouses, serverless notebooks and jobs, and Lakeflow pipelines, so you can measure how often people use governed metric views instead of raw tables. That evidence turns governance from a policy document into something you can track weekly.
    Two recent changes affect the setup. Since August 2026, the statement_text column is masked as <REDACTED> by default, and only account admins and members of the databricks_pii_access group can read full query text, so grant that access deliberately to the platform team that runs the review. The query_tags and query_source columns help too, because they let you slice usage by team, dashboard or Genie Agent without parsing SQL.
    An insurer reviewing query history each week might find claims dashboards still hitting a legacy table months after the governed metric launched. That finding points the platform team at specific workbook owners instead of a broad reminder email. These are the signals worth reviewing every week.
    • Share of BI queries that reference metric views rather than raw gold tables
    • Users or service principals still querying deprecated objects
    • Repeated ad hoc queries that point to a missing governed metric
    • Warehouse queue time during peak dashboard refresh windows, from the waiting_at_capacity_duration_ms column
    • Failed queries that follow a change to a metric definition
    • Dashboards and Genie Agents, identified through query_source, that still bypass the governed layer

    Why trusted metrics depend on disciplined ownership over time

    Trusted analytics come from a small number of well-owned definitions that people keep using because they keep being right. Databricks gives you the tooling to make that possible, but the trust itself is earned one quarter at a time, through review discipline, clear ownership and visible evidence of use.
    The platform advantage is real, and it's worth naming. On Databricks, tables, metrics, dashboards, models and agents share one permission model, one lineage graph and one audit trail, so there's no extract to keep in sync and no second governance system to reconcile. That shared foundation is what lets a metric defined for the finance dashboard also answer the CFO's question in Genie One without a separate build.
    The teams that get this right treat a metric like a product with a release process. They retire duplicate definitions deliberately, publish change notes when logic shifts and check query history before declaring a migration done. Each habit is small. Combined, they're what keeps a finance leader from rebuilding the number in a spreadsheet.
    Start narrower than feels comfortable. Ten governed metrics that every executive dashboard reads will build more confidence than 200 that nobody has validated. Once those hold up through a few close cycles, expanding the layer gets easier because people already believe the numbers.
    Lumenalta®, a Silver-tier partner in the Databricks Brickbuilder Partner Program with Databricks MVPs and Champions on the team, builds semantic layers this way, starting with the metrics leadership argues about most and extending from there. We bring the business semantics, Genie and governance releases into client roadmaps as they mature, so you adopt new capabilities when they're ready rather than when they're announced. You'll know the layer is working when the quarterly review spends its time on what the numbers mean instead of debating if they're correct.
    Table of contents
    See how a semantic layer on Databricks lowers cost and improves data agility.