From Legacy EDW to a Modern Lakehouse: Lessons from Databricks’ EDW-ETL Migration Course

“So, how’s it going with Databricks? I’ve heard about it here and there, but it seems like a lot has changed, right?”

That’s the question a colleague asked me a couple of days ago. Today, while going through the “EDW-ETL Migration to the Data Intelligence Platform Partner Presales Badge” course, I had the chance to reflect more on the topic.

Databricks Then and Now

What were “Databricks” in the past, and what are “Databricks” now? According to Databricks materials, 74% of enterprises have already deployed a Lakehouse architecture. Meanwhile, I’ve spent 15 years in data — much of it with SQL Server — and last year, I became deeply active in the Databricks Community. That transition gives me a perspective I underestimated when I was younger.

• I started my journey with SQL Server at a time when SSIS, SSRS, and SSAS fulfilled most BI needs, delivering “classic” data warehouse solutions.
• About seven years ago, the focus shifted to cloud data lakes (often jokingly called data swamps).
• Four years ago, I began using Databricks. It wasn’t smooth sailing initially, and I faced a few key hurdles:

1. Cluster Start Time