Hi there Of us!
Welcome again to the ITOpsTalk Weblog and the Microsoft Azure Infrastructure Summit 2026 sequence. On this session James Baker and Sai Runtham, each from the Azure Information Lake Storage product crew, take us by way of what a contemporary Lakehouse really is, tips on how to design one on Azure, after which they roll up their sleeves and construct one finish to finish. You probably have been listening to “Lakehouse” thrown round in structure critiques and weren’t 100% positive what it adjustments for you as an IT Professional, this one is for you.
📺 Watch the session:
You is likely to be pondering, “I run infrastructure, not analytics.” Honest level. However right here is the factor. The lakehouse is more and more the platform your corporation will run BI dashboards, AI brokers, and determination help methods on, and you’re the one who has to maintain the info secure, ruled, and reachable.
Here’s what is in it for you:
- It’s a platform dialog. James spends an enormous chunk of the session on horizontal platform capabilities (storage, catalog, id, secrets and techniques, community, coverage) versus vertical pipeline issues. That’s squarely an IT Professional downside.
- Information is the asset. Workspaces, question engines, and dashboards are transient. The information lives endlessly, and defending it’s in your plate.
- Governance is what stops your knowledge lake from rotting into a knowledge swamp.
- Scale is a virtuous cycle. Extra knowledge drives extra perception, which drives extra knowledge. Your platform can’t develop into the ceiling.
- AI brokers are the brand new customers. They don’t simply learn dashboards, they question gold tables straight. Your community, id, and entry controls need to sustain.
A knowledge lakehouse is precisely what it feels like. You are taking a budget, versatile, schema-light scale of a knowledge lake, and also you fuse it with the low-latency question efficiency, replace semantics, and governance of a knowledge warehouse. One copy of the info. One place to manipulate it. No extra forking from the lake right into a warehouse simply to make BI instruments glad.
Fast distinction:
- Information lake. Large, low-cost, versatile. No schema enforced on write. Traditionally vulnerable to turning into a swamp.
- Information warehouse. Low-latency queries, updates, sturdy governance, structured. Hits a scale ceiling and prices extra.
- Information lakehouse. Lake-scale storage, with a high-performance question layer and warehouse-grade governance sitting excessive. No knowledge fork.
The massive shift is that the info doesn’t transfer. Your BI dashboards, your AI brokers, your serverless SQL queries, all of them hit the identical ruled tables within the lake. That retains lineage clear and your safety mannequin sane.
James and Sai are clear that the structure is much less a hard and fast diagram and extra a listing of platform capabilities you compose. Right here is the form of it on Azure.
Storage layer (the asset).
- Azure Information Lake Storage Gen2 (ADLS) with hierarchical namespace turned on. That’s non-negotiable for analytics workloads. It offers you atomic listing operations, POSIX-style ACLs, and the efficiency Delta Lake depends on.
- OneLake in Microsoft Cloth if you would like a tenant-wide logical lake that’s constructed on ADLS Gen2 and shops all the things in open Delta Parquet by default.
Desk format and pipelines.
- Open desk codecs: Delta Lake (and Apache Iceberg because it converges) provide you with ACID transactions, time journey, schema evolution, and streaming on low-cost object storage.
- Azure Databricks Lakeflow Declarative Pipelines with Autoloader for incremental ingestion of each batch and streaming sources straight into Delta tables. Autoloader handles new file discovery, schema inference, and evolution for you.
- The medallion structure for stamping out repeatable pipelines:
o Bronze. Uncooked, append-only touchdown zone. Supply of reality.
o Silver. Cleansed, deduplicated, conformed, enriched.
o Gold. Enterprise-ready, aggregated, performance-optimized for consumption.
Governance and id.
- Unity Catalog as the only supply of reality for catalog, lineage, and fine-grained entry management throughout bronze, silver, and gold.
- Entra ID for id. Managed identities for compute. Key Vault for secrets and techniques.
- Community safety across the perimeter. The information is the crown jewel, so personal endpoints, firewalls, and VNet-attached compute are baseline.
Consumption layer.
- Energy BI Direct Question towards a serverless SQL warehouse on the gold tables. No knowledge copies, governance flows by way of.
- AI brokers like Databricks Genie pointed at gold tables. Pure-language questions, dwell lineage, no knowledge motion.
The demo that ties it collectively. Sai walked by way of an actual pipeline: NYC TLC taxi journeys enriched with NOAA climate and ESPN/MLB sports activities occasions, ingested by Autoloader into bronze, remodeled by way of silver, aggregated into gold. A parallel streaming pipeline dealt with artificial dwell occasions for a real-time demand view. Energy BI dashboards hit gold by way of Direct Question. And Genie answered questions like “which zones are most delicate to sport occasions” by mapping demand round Madison Sq. Backyard, with the question and the chart generated for you. All towards the identical lakehouse, no knowledge motion, full lineage.
That is the place a whole lot of lakehouses go sideways. A couple of issues from the session and from the official steering price pinning to your wall:
- Get hierarchical namespace proper. It’s the distinction between atomic listing operations and “copy then delete,” which is gradual and costly at scale.
- Use storage tiers and lifecycle insurance policies. Sizzling for working knowledge, Cool or Chilly for older partitions, Archive for compliance retention. Lifecycle guidelines on ADLS do that routinely.
- Partition and file-size matter. Numerous tiny information kill question efficiency. Use OPTIMIZE, Z-Order, or liquid clustering on Delta tables, and partition on the columns your queries really filter on.
- Lean on vectorized reads. ADLS plus Delta plus fashionable question engines push a whole lot of work right down to columnar Parquet, which retains your compute invoice in test.
- Use serverless SQL warehouses the place it matches. Direct Question towards a serverless endpoint scales compute to demand and allows you to maintain dashboards contemporary with out import refreshes.
- Observe knowledge, not simply methods. “Is Databricks up” is critical however not ample. Watch knowledge freshness, row counts, pipeline blockages, and SLAs on the info itself.
- Govern all the things. A well-governed lakehouse drives belief, which drives use, which drives worth. Skipping governance early at all times prices extra later.
If you wish to put palms on a keyboard this week:
- Spin up an Azure Storage account with hierarchical namespace enabled. That’s your ADLS Gen2 basis.
- Get up an Azure Databricks workspace, allow Unity Catalog, and level it at your ADLS account.
- Create a Lakeflow Declarative Pipeline. Use Autoloader to ingest a pattern dataset (the NYC taxi knowledge is a traditional start line) right into a bronze Delta desk.
- Add silver and gold notebooks or pipelines that clear and mixture the info.
- Join Energy BI to a serverless SQL warehouse in your gold tables with Direct Question.
- If you’re a Cloth tenant, mirror or shortcut knowledge into OneLake and check out the identical sample there, no infra to handle.
- Learn the Hitchhiker’s Information to ADLS earlier than you scale up. It can prevent future you a whole lot of grief.
This session is one cease on an enormous tour. The total Microsoft Azure Infrastructure Summit 2026 playlist covers all the things from sovereign cloud and AKS networking to backup, storage, and AI-assisted operations. In case your job touches Azure, there’s something in right here for you.
Head over to the total playlist and binge what is beneficial: https://www.youtube.com/playlist?record=PLjt5SKzX1iI8con7FJDB56G6hHqxGm7ki
Cheers!
Pierre Roman
